{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 PyTorch第一步\n",
    "\n",
    "PyTorch的简洁设计使得它入门很简单，在深入介绍PyTorch之前，本节将先介绍一些PyTorch的基础知识，使得读者能够对PyTorch有一个大致的了解，并能够用PyTorch搭建一个简单的神经网络。部分内容读者可能暂时不太理解，可先不予以深究，本书的第3章和第4章将会对此进行深入讲解。\n",
    "\n",
    "本节内容参考了PyTorch官方教程[^1]并做了相应的增删修改，使得内容更贴合新版本的PyTorch接口，同时也更适合新手快速入门。另外本书需要读者先掌握基础的Numpy使用，其他相关知识推荐读者参考CS231n的教程[^2]。\n",
    "\n",
    "[^1]: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n",
    "[^2]: http://cs231n.github.io/python-numpy-tutorial/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor\n",
    "\n",
    "Tensor是PyTorch中重要的数据结构，可认为是一个高维数组。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）以及更高维的数组。Tensor和Numpy的ndarrays类似，但Tensor可以使用GPU进行加速。Tensor的使用和Numpy及Matlab的接口十分相似，下面通过几个例子来看看Tensor的基本使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.4.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " from __future__ import print_function\n",
    "import torch as t\n",
    "t.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  2.],\n",
       "        [ 3.,  4.]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 构建 5x3 矩阵，只是分配了空间，未初始化\n",
    "x = t.Tensor(5, 3)\n",
    "\n",
    "x = t.Tensor([[1,2],[3,4]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.8052,  0.7188,  0.0332],\n",
       "        [ 0.6054,  0.8955,  0.8972],\n",
       "        [ 0.1107,  0.3319,  0.0336],\n",
       "        [ 0.2394,  0.5188,  0.2201],\n",
       "        [ 0.9730,  0.9370,  0.5677]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用[0,1]均匀分布随机初始化二维数组\n",
    "x = t.rand(5, 3)  \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 3])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(x.size()) # 查看x的形状\n",
    "x.size()[1], x.size(1) # 查看列的个数, 两种写法等价"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.Size` 是tuple对象的子类，因此它支持tuple的所有操作，如x.size()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9639,  0.8763,  0.2834],\n",
       "        [ 1.3785,  1.5090,  1.3919],\n",
       "        [ 0.7139,  0.6348,  0.8439],\n",
       "        [ 0.7022,  1.5079,  0.4776],\n",
       "        [ 1.7892,  1.6383,  0.7774]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = t.rand(5, 3)\n",
    "# 加法的第一种写法\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9639,  0.8763,  0.2834],\n",
       "        [ 1.3785,  1.5090,  1.3919],\n",
       "        [ 0.7139,  0.6348,  0.8439],\n",
       "        [ 0.7022,  1.5079,  0.4776],\n",
       "        [ 1.7892,  1.6383,  0.7774]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第二种写法\n",
    "t.add(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9639,  0.8763,  0.2834],\n",
       "        [ 1.3785,  1.5090,  1.3919],\n",
       "        [ 0.7139,  0.6348,  0.8439],\n",
       "        [ 0.7022,  1.5079,  0.4776],\n",
       "        [ 1.7892,  1.6383,  0.7774]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第三种写法：指定加法结果的输出目标为result\n",
    "result = t.Tensor(5, 3) # 预先分配空间\n",
    "t.add(x, y, out=result) # 输入到result\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "最初y\n",
      "tensor([[ 0.1587,  0.1575,  0.2501],\n",
      "        [ 0.7732,  0.6135,  0.4947],\n",
      "        [ 0.6033,  0.3029,  0.8103],\n",
      "        [ 0.4628,  0.9891,  0.2575],\n",
      "        [ 0.8163,  0.7013,  0.2097]])\n",
      "第一种加法，y的结果\n",
      "tensor([[ 0.1587,  0.1575,  0.2501],\n",
      "        [ 0.7732,  0.6135,  0.4947],\n",
      "        [ 0.6033,  0.3029,  0.8103],\n",
      "        [ 0.4628,  0.9891,  0.2575],\n",
      "        [ 0.8163,  0.7013,  0.2097]])\n",
      "第二种加法，y的结果\n",
      "tensor([[ 0.9639,  0.8763,  0.2834],\n",
      "        [ 1.3785,  1.5090,  1.3919],\n",
      "        [ 0.7139,  0.6348,  0.8439],\n",
      "        [ 0.7022,  1.5079,  0.4776],\n",
      "        [ 1.7892,  1.6383,  0.7774]])\n"
     ]
    }
   ],
   "source": [
    "print('最初y')\n",
    "print(y)\n",
    "\n",
    "print('第一种加法，y的结果')\n",
    "y.add(x) # 普通加法，不改变y的内容\n",
    "print(y)\n",
    "\n",
    "print('第二种加法，y的结果')\n",
    "y.add_(x) # inplace 加法，y变了\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，函数名后面带下划线**`_`** 的函数会修改Tensor本身。例如，`x.add_(y)`和`x.t_()`会改变 `x`，但`x.add(y)`和`x.t()`返回一个新的Tensor， 而`x`不变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.7188,  0.8955,  0.3319,  0.5188,  0.9370])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Tensor的选取操作与Numpy类似\n",
    "x[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor还支持很多操作，包括数学运算、线性代数、选择、切片等等，其接口设计与Numpy极为相似。更详细的使用方法，会在第三章系统讲解。\n",
    "\n",
    "Tensor和Numpy的数组之间的互操作非常容易且快速。对于Tensor不支持的操作，可以先转为Numpy数组处理，之后再转回Tensor。c\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.,  1.,  1.,  1.,  1.])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.ones(5) # 新建一个全1的Tensor\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.numpy() # Tensor -> Numpy\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1. 1. 1. 1.]\n",
      "tensor([ 1.,  1.,  1.,  1.,  1.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones(5)\n",
    "b = t.from_numpy(a) # Numpy->Tensor\n",
    "print(a)\n",
    "print(b) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor和numpy对象共享内存，所以他们之间的转换很快，而且几乎不会消耗什么资源。但这也意味着，如果其中一个变了，另外一个也会随之改变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2. 2. 2. 2. 2.]\n",
      "tensor([ 2.,  2.,  2.,  2.,  2.], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "b.add_(1) # 以`_`结尾的函数会修改自身\n",
    "print(a)\n",
    "print(b) # Tensor和Numpy共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你想获取某一个元素的值，可以使用`scalar.item`。 直接`tensor[idx]`得到的还是一个tensor: 一个0-dim 的tensor，一般称为scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2., dtype=torch.float64)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar = b[0]\n",
    "scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar.size() #0-dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar.item() # 使用scalar.item()能从中取出python对象的数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 2]), tensor(2., dtype=torch.float64))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = t.tensor([2]) # 注意和scalar的区别\n",
    "tensor,scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1]), torch.Size([]))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor.size(),scalar.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 2.0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 只有一个元素的tensor也可以调用`tensor.item()`\n",
    "tensor.item(), scalar.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外在pytorch中还有一个和`np.array` 很类似的接口: `torch.tensor`, 二者的使用十分类似。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = t.tensor([3,4]) # 新建一个包含 3，4 两个元素的tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalar = t.tensor(3)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 3,  4]), tensor([ 1111,     4]))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_tensor = tensor\n",
    "new_tensor = t.tensor(old_tensor)\n",
    "new_tensor[0] = 1111\n",
    "old_tensor, new_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，`t.tensor()`总是会进行数据拷贝，新tensor和原来的数据不再共享内存。所以如果你想共享内存的话，建议使用`torch.from_numpy()`或者`tensor.detach()`来新建一个tensor, 二者共享内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 1111,     4]), tensor([ 1111,     4]))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_tensor = old_tensor.detach()\n",
    "new_tensor[0] = 1111\n",
    "old_tensor, new_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor可通过`.cuda` 方法转为GPU的Tensor，从而享受GPU带来的加速运算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在不支持CUDA的机器下，下一步还是在CPU上运行\n",
    "device = t.device(\"cuda:0\" if t.cuda.is_available() else \"cpu\")\n",
    "x = x.to(device)\n",
    "y = y.to(device)\n",
    "z = x+y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，还可以使用`tensor.cuda()` 的方式将tensor拷贝到gpu上，但是这种方式不太推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处可能发现GPU运算的速度并未提升太多，这是因为x和y太小且运算也较为简单，而且将数据从内存转移到显存还需要花费额外的开销。GPU的优势需在大规模数据和复杂运算下才能体现出来。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### autograd: 自动微分\n",
    "\n",
    "深度学习的算法本质上是通过反向传播求导数，而PyTorch的**`autograd`**模块则实现了此功能。在Tensor上的所有操作，autograd都能为它们自动提供微分，避免了手动计算导数的复杂过程。\n",
    " \n",
    "~~`autograd.Variable`是Autograd中的核心类，它简单封装了Tensor，并支持几乎所有Tensor有的操作。Tensor在被封装为Variable之后，可以调用它的`.backward`实现反向传播，自动计算所有梯度~~ ~~Variable的数据结构如图2-6所示。~~\n",
    "\n",
    "\n",
    "![图2-6:Variable的数据结构](imgs/autograd_Variable.svg)\n",
    "\n",
    "  *从0.4起, Variable 正式合并入Tensor, Variable 本来实现的自动微分功能，Tensor就能支持。读者还是可以使用Variable(tensor), 但是这个操作其实什么都没做。建议读者以后直接使用tensor*. \n",
    "  \n",
    "  要想使得Tensor使用autograd功能，只需要设置`tensor.requries_grad=True`. \n",
    "\n",
    "\n",
    "~~Variable主要包含三个属性。~~\n",
    "~~- `data`：保存Variable所包含的Tensor~~\n",
    "~~- `grad`：保存`data`对应的梯度，`grad`也是个Variable，而不是Tensor，它和`data`的形状一样。~~\n",
    "~~- `grad_fn`：指向一个`Function`对象，这个`Function`用来反向传播计算输入的梯度，具体细节会在下一章讲解。~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.],\n",
       "        [ 1.,  1.]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为tensor设置 requires_grad 标识，代表着需要求导数\n",
    "# pytorch 会自动调用autograd 记录操作\n",
    "x = t.ones(2, 2, requires_grad=True)\n",
    "\n",
    "# 上一步等价于\n",
    "# x = t.ones(2,2)\n",
    "# x.requires_grad = True\n",
    "\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4.)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.sum()\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<SumBackward0 at 0x7ffaa589a780>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward() # 反向传播,计算梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.],\n",
       "        [ 1.,  1.]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# y = x.sum() = (x[0][0] + x[0][1] + x[1][0] + x[1][1])\n",
    "# 每个值的梯度都为1\n",
    "x.grad "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：`grad`在反向传播过程中是累加的(accumulated)，这意味着每一次运行反向传播，梯度都会累加之前的梯度，所以反向传播之前需把梯度清零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 2.,  2.],\n",
       "        [ 2.,  2.]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 3.,  3.],\n",
       "        [ 3.,  3.]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.],\n",
       "        [ 0.,  0.]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以下划线结束的函数是inplace操作，会修改自身的值，就像add_\n",
    "x.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.],\n",
       "        [ 1.,  1.]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  神经网络\n",
    "\n",
    "Autograd实现了反向传播功能，但是直接用来写深度学习的代码在很多情况下还是稍显复杂，torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。nn.Module是nn中最重要的类，可把它看成是一个网络的封装，包含网络各层定义以及forward方法，调用forward(input)方法，可返回前向传播的结果。下面就以最早的卷积神经网络：LeNet为例，来看看如何用`nn.Module`实现。LeNet的网络结构如图2-7所示。\n",
    "\n",
    "![图2-7:LeNet网络结构](imgs/nn_lenet.png)\n",
    "\n",
    "这是一个基础的前向传播(feed-forward)网络: 接收输入，经过层层传递运算，得到输出。\n",
    "\n",
    "#### 定义网络\n",
    "\n",
    "定义网络时，需要继承`nn.Module`，并实现它的forward方法，把网络中具有可学习参数的层放在构造函数`__init__`中。如果某一层(如ReLU)不具有可学习的参数，则既可以放在构造函数中，也可以不放，但建议不放在其中，而在forward中使用`nn.functional`代替。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        # nn.Module子类的函数必须在构造函数中执行父类的构造函数\n",
    "        # 下式等价于nn.Module.__init__(self)\n",
    "        super(Net, self).__init__()\n",
    "        \n",
    "        # 卷积层 '1'表示输入图片为单通道, '6'表示输出通道数，'5'表示卷积核为5*5\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5) \n",
    "        # 卷积层\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5) \n",
    "        # 仿射层/全连接层，y = Wx + b\n",
    "        self.fc1   = nn.Linear(16*5*5, 120) \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        # 卷积 -> 激活 -> 池化 \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        # reshape，‘-1’表示自适应\n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只要在nn.Module的子类中定义了forward函数，backward函数就会自动被实现(利用`autograd`)。在`forward` 函数中可使用任何tensor支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。\n",
    "\n",
    "网络的可学习参数通过`net.parameters()`返回，`net.named_parameters`可同时返回可学习的参数及名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.weight : torch.Size([6, 1, 5, 5])\n",
      "conv1.bias : torch.Size([6])\n",
      "conv2.weight : torch.Size([16, 6, 5, 5])\n",
      "conv2.bias : torch.Size([16])\n",
      "fc1.weight : torch.Size([120, 400])\n",
      "fc1.bias : torch.Size([120])\n",
      "fc2.weight : torch.Size([84, 120])\n",
      "fc2.bias : torch.Size([84])\n",
      "fc3.weight : torch.Size([10, 84])\n",
      "fc3.bias : torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "for name,parameters in net.named_parameters():\n",
    "    print(name,':',parameters.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "forward函数的输入和输出都是Tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = t.randn(1, 1, 32, 32)\n",
    "out = net(input)\n",
    "out.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad() # 所有参数的梯度清零\n",
    "out.backward(t.ones(1,10)) # 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，torch.nn只支持mini-batches，不支持一次只输入一个样本，即一次必须是一个batch。但如果只想输入一个样本，则用 `input.unsqueeze(0)`将batch_size设为１。例如 `nn.Conv2d` 输入必须是4维的，形如$nSamples \\times nChannels \\times Height \\times Width$。可将nSample设为1，即$1 \\times nChannels \\times Height \\times Width$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 损失函数\n",
    "\n",
    "nn实现了神经网络中大多数的损失函数，例如nn.MSELoss用来计算均方误差，nn.CrossEntropyLoss用来计算交叉熵损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(28.5625)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = net(input)\n",
    "target = t.arange(0,10).view(1,10) \n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(output, target)\n",
    "loss # loss是个scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果对loss进行反向传播溯源(使用`gradfn`属性)，可看到它的计算图如下：\n",
    "\n",
    "```\n",
    "input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d  \n",
    "      -> view -> linear -> relu -> linear -> relu -> linear \n",
    "      -> MSELoss\n",
    "      -> loss\n",
    "```\n",
    "\n",
    "当调用`loss.backward()`时，该图会动态生成并自动微分，也即会自动计算图中参数(Parameter)的导数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "反向传播之前 conv1.bias的梯度\n",
      "tensor([ 0.,  0.,  0.,  0.,  0.,  0.])\n",
      "反向传播之后 conv1.bias的梯度\n",
      "tensor(1.00000e-02 *\n",
      "       [-2.2701, -1.8687,  4.3639, -2.9262, -4.3535, -5.9584])\n"
     ]
    }
   ],
   "source": [
    "# 运行.backward，观察调用之前和调用之后的grad\n",
    "net.zero_grad() # 把net中所有可学习参数的梯度清零\n",
    "print('反向传播之前 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)\n",
    "loss.backward()\n",
    "print('反向传播之后 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 优化器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播计算完所有参数的梯度后，还需要使用优化方法来更新网络的权重和参数，例如随机梯度下降法(SGD)的更新策略如下：\n",
    "```\n",
    "weight = weight - learning_rate * gradient\n",
    "```\n",
    "\n",
    "手动实现如下：\n",
    "\n",
    "```python\n",
    "learning_rate = 0.01\n",
    "for f in net.parameters():\n",
    "    f.data.sub_(f.grad.data * learning_rate)# inplace 减法\n",
    "```\n",
    "\n",
    "`torch.optim`中实现了深度学习中绝大多数的优化方法，例如RMSProp、Adam、SGD等，更便于使用，因此大多数时候并不需要手动写上述代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "#新建一个优化器，指定要调整的参数和学习率\n",
    "optimizer = optim.SGD(net.parameters(), lr = 0.01)\n",
    "\n",
    "# 在训练过程中\n",
    "# 先梯度清零(与net.zero_grad()效果一样)\n",
    "optimizer.zero_grad() \n",
    "\n",
    "# 计算损失\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "\n",
    "#反向传播\n",
    "loss.backward()\n",
    "\n",
    "#更新参数\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "####  数据加载与预处理\n",
    "\n",
    "在深度学习中数据加载及预处理是非常复杂繁琐的，但PyTorch提供了一些可极大简化和加快数据处理流程的工具。同时，对于常用的数据集，PyTorch也提供了封装好的接口供用户快速调用，这些数据集主要保存在torchvison中。\n",
    "\n",
    "`torchvision`实现了常用的图像数据加载功能，例如Imagenet、CIFAR10、MNIST等，以及常用的数据转换操作，这极大地方便了数据加载，并且代码具有可重用性。\n",
    "\n",
    "\n",
    "### 小试牛刀：CIFAR-10分类\n",
    "\n",
    "下面我们来尝试实现对CIFAR-10数据集的分类，步骤如下: \n",
    "\n",
    "1. 使用torchvision加载并预处理CIFAR-10数据集\n",
    "2. 定义网络\n",
    "3. 定义损失函数和优化器\n",
    "4. 训练网络并更新网络参数\n",
    "5. 测试网络\n",
    "\n",
    "####   CIFAR-10数据加载及预处理\n",
    "\n",
    "CIFAR-10[^3]是一个常用的彩色图片数据集，它有10个类别: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'。每张图片都是$3\\times32\\times32$，也即3-通道彩色图片，分辨率为$32\\times32$。\n",
    "\n",
    "[^3]: http://www.cs.toronto.edu/~kriz/cifar.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision as tv\n",
    "import torchvision.transforms as transforms\n",
    "from torchvision.transforms import ToPILImage\n",
    "show = ToPILImage() # 可以把Tensor转成Image，方便可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# 第一次运行程序torchvision会自动下载CIFAR-10数据集，\n",
    "# 大约100M，需花费一定的时间，\n",
    "# 如果已经下载有CIFAR-10，可通过root参数指定\n",
    "\n",
    "# 定义对数据的预处理\n",
    "transform = transforms.Compose([\n",
    "        transforms.ToTensor(), # 转为Tensor\n",
    "        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # 归一化\n",
    "                             ])\n",
    "\n",
    "# 训练集\n",
    "trainset = tv.datasets.CIFAR10(\n",
    "                    root='/home/cy/tmp/data/', \n",
    "                    train=True, \n",
    "                    download=True,\n",
    "                    transform=transform)\n",
    "\n",
    "trainloader = t.utils.data.DataLoader(\n",
    "                    trainset, \n",
    "                    batch_size=4,\n",
    "                    shuffle=True, \n",
    "                    num_workers=2)\n",
    "\n",
    "# 测试集\n",
    "testset = tv.datasets.CIFAR10(\n",
    "                    '/home/cy/tmp/data/',\n",
    "                    train=False, \n",
    "                    download=True, \n",
    "                    transform=transform)\n",
    "\n",
    "testloader = t.utils.data.DataLoader(\n",
    "                    testset,\n",
    "                    batch_size=4, \n",
    "                    shuffle=False,\n",
    "                    num_workers=2)\n",
    "\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset对象是一个数据集，可以按下标访问，返回形如(data, label)的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ship\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAIAAAD/gAIDAAALVElEQVR4nO1cW3MVxxGe2d1zk46E\nhEASlpCEBaEo43K5UqmUKz8jpCo/MQ/Jj0j5JQllLGIw2NxsK4iLERJH17PXPEz316OZxdLoeb4X\nWruzvbPDfNOX6Tn64cuRUkopVde1OomqEbmsaqcZhIKbFTVJVVV5jRsWRGdRlaRc8d2Gbmtu3/AD\nTdM4Ql4m0tXavYs+NI1mVepjX9pUckUXlXMX7RMVcWbEwQpAppkCEACttMgsJiw1uOK1EYHvJWht\nvQWqUhY0s0FrJqbGYy5V00S6Bwjf5RpdSZKU/vaorRrpldau2oRfFGdWAOJgBSBLZMLy5OS/Ey1D\nCakRXqAZBDZJunEayRVrkguNEpe3iSwO3DkxWCCv9R0ed1J0hvsO9uFtYLTSNg3d5QhsjTMrAHGw\nApBZfHJnvj2zwYtaYTLz5GzQ5qQiS2w8U6tsHmm3WeL1wmK9e+VEJ+C7ijlkN5VvZQluJM5HqTbv\nF0Y6zqwAxMEKQBysAGRwWH0XO7GMqCxVWNr4AawFvivQ1O6CaLvdWpNLXXNEnYgn7boC0r0Ga6ul\nSvrAjgVPg6pkj58vQUOt8S2W68APIhiIHvx5EAcrAJk18wkWc+ygF3EsLnLuSey9qwNudC0+hJU5\nEp/Ddf0tEvqeDFwHiztYHBpXgzg0csv3VERV43EzevDnQRysAEg+S8JgvldbbEgSmBKka+mWzHyh\nFWJyNIJOC4hs4VJLPotttJDPzZbZy0cj7V2zqFJXuZh7lmp7zqAPSMYpV4g4HXGwAiCBdIt32tjW\nENlk7d/FE87f2mOHnSxqGtCQA1rtPqi8KxBOmjD0wPOQvTy45fm2JA1ASY3eeHyMOB1xsAKQiY/n\n08q68BteX4vQss9DQm3lfxEb4lErGhVb62mSXLXVP5ek1h0v2e0pP/HtkuJ2I9w4swIQBysAWcPT\nrvIqAM6I1CcdiMIzuYBHmGR4MEFWl1mQ8pNlUzhv0QolCOzxCotV3fD/OvIwKLNgVbVmd9oLRWtR\nLouD1q6vHGdWAOJgBUBI0VKMcDYknt8olUaNn8axd114ejeunRKC1O5mkrWDeyI6dL7DN82+oYQ1\nTKz8rV8A4Wd7Ik5HHKwAZFq5+/26JegTaEmQIg/jjnjLhEey0dr8hJXBbgjebO3XukkebM3aqkBk\nLQkc3PXKJrxMqd3hunYXkxQ+s/tVER9HHKwAZLLm+5nEVrRUyzGJavdOW+7UripgS4dSPNGIJCZt\nKaZI47CG1OonzC7KpPx9B+isGrwOHRUaViBpTZ5qmqb8FRFnRhysAMTBCoCUSYK24H9b4liWNvHF\nUZXfkrjlhYP5n1mrQ8aBcCUmPOFuUbMclf68q4QOpFYVJ+JorINwgLAdlXiOQtW6THsZ7RhInwdx\nsAJgbbJaRbzm38r2tgE/Nhan2fWMMZNB0IP9D9C0vf3OCEXB2StW1ZuYcl47nBxSr/hQTpL15TO4\n82VJroZfoCDuiFecUdvBAF/WnBqLZ3fOgzhYAcjatm1cwYb2JrNVe8R/s+1Dk4QLfp/98BCq7t69\na4TxeGyEPCc+Fg1Zyi++/NIIn9++bQTQcHK2B1U4QqekNApW3k1tV6UbFdi5A1hPmGZvszXiDIiD\nFYAsscp86N/W3BPguay1xmRmTR5/Gy6xXbh0ERdXlz+hFzEdtt+/N0JeEw0zVvr4+wdGuH79Bt86\n8Qb+CPSKbTrTFoF3guJcvlLZpcbMOkmXt5Q2RpyGOFgBaNndCd/fYQ1ylo6Jyf8X+TGZuV5X3njz\nxroRpqbIBf3mm3tG6A5njXBwdER9YtZfnL3g99M6g4cKRWTZvEIoT0r8PLhStXcsPs6sAMTBCoCY\nlMqLqqQw1vb6pJCBnTdVOQ+CAjjR8fbtKyN8d/9b6Dw+PjbC5i+/GCHNiKTXrpOw9XLLCF999Sfu\nFPWqKqQeIvUOi9f8OR22ffiZCvldB8mQW7UOqPzDOHBqO86sAMTBCkBWeT+XIlV6lt2QX3GQ/U9q\nX1aF00YOjLEvOneZrJvqiDVMFQV3U3Nz1GyOXNa8yo2w9YpoOL+wyMq5JMi22rUwivopd9wtnFq5\nYeOJPSfv5EyTRGsYjjhYAcgQOllzklDVYiPQLFMwgkix8hlLsaL0f3BhetoIPzx5YoT5K8vQeXBw\nYISpGaLh/v6+EV5vEfuevPjJCH/7+z+M8Jc7fzVCryuZUuvnlOhKXoBE2hFg2cUVtew+fNESzWKt\nwzkQBysAcbACkB0XpXNJ9kUsM4/cccXubJmT/52mXW5BQ//zTz8b4e3bX42wf3hohPxEJRScD96w\n6Q2MsLh01QhXr103wmBIy193YpJ7YvWZ/Ymyoe6N+St6aYe/y1udJeQQVVhwk9oNSOLMCkAcrABk\n9+7/10jwtuEldKzcU6/DfnNN/vrkgPzvJCEaNglduXdvwwgbG/eNsLu3Z4SF1TXoXF4mN+Lp06dG\nmGNXfmVlxQjrN24aYW2Nkl9vft02wrgQHoJZ45w2ipBTyziQxg6TtfdLRCtKey1q4SZpcC9EfBxx\nsAKQvf+wa6TBgCxRxkmlzLKGmoPJNSbIzDTlgvsDqkJ49uJ/dGuGMr/r69eMsDMi13x6fhE6//Xv\n/xhhc3PTCCWnqO7c+bMRZmcptH786LER3rwmGua2OWQTdshmt9MhIwinPpX9Hg6k4dNbNMTeKtYl\nv4Y64nTEwQpABpNSHNAEnp2l3FOv30W7hUt0scPcHI12jbC3T/Gw4jNqv7tJlmtpiUi3u0c03DnM\nofOPf/i9Eb74/DNqtks6+/zqmRnyRY8OaJvnYH/EfWeiWdVRiIgrzohhdwe0bbyAv2yj4W9UL0Wc\njjhYAcgSnszb22Rl9njCPzvaQbseVwpcmiVepFLaQCPe53I9mNGq5NxQ2bJBsrJ8hVRxVT4MMRzj\nfEz28ZPFy0bY3KRUV29yILqYUKMRkTTPmYZcnIsMV8qVvzCCRdFCQ+tcbsxnhSMOVgCyhmfdxUs0\nz1EOW42lWLbhY9mDASVzUQePCp5KUZuDQ7KPBVfyjXMOPGsxYTnzGDSE3cmYKSknWLocga6vXnUe\nV0qV7HlWnDhqeM8JDNOpe1K8kjNDkjgqeenAmlDHFM05EAcrABkog1mHdAccQqWULjkvyns5OVfN\n9jPKzHSEO8je8OOY+aX1Yww1NjvlPdyM+ctv2d+jDmRMzP60dC/nOG5+boaUF2TT9yoUPXT4HbKB\nRVcSoXQxphdVXAQMWxlnVgDiYAUgO2YaznEyBDwBv5RSyyuU1ex1aTI/evS9EV5uvTHCYEhbCUh4\ndlLyG3WXnUxl5yS50LxyDWuGA6mcGtIDEsbwNot9UcQBYMo1VDOTE0Y4PqRDL3VO2VosF3ND3h9Z\nmIcq1Dq8eU0PVtXgRHcjzoI4WAGIgxWAbOEy0fWIyzQS9iFu3/4M7VaWKTO1NyLmT0xQNvnwmIz0\n0xfPjfDkx2eknVUhRzbJJ+GU5a9P8PrS4aieM2MSig/6tHCguPKoOIYq/KbTaIeC//l5itKHvJIO\np+gtV68sGGHpCn17t2M5NLwX++7dB/5k+sA4swIQBysAGfI+MMljrtPf2JDK4offkYBULJJWq2tr\nRrh165YRUGb14AEduHn+nBi6s7MLnb0eu/68EwNh0KFb3Q7Fz91u12lTWbWNSUqdQeHFCgf8K4ur\nRri6St7PBU6E9bFzbKnCNm2vR+m50ZAS7nFmBSAOVgAyJGum+QDN+JBouPVqE+0O93aNAIp1mBf/\n/PprI3Q9WoE7S0tLRsjzH6ETaazhkExkxldqjl1hm0bcAcTkCJ6VUkfHtIZ8yiVKO2wWYaw7XVI+\n9SkRM0mQ/hYavt+mF/X7ZD3n5siUx5kVgDhYAfg/pQ4eZ65sAxcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=100x100 at 0x7FFA9E669DA0>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(data, label) = trainset[100]\n",
    "print(classes[label])\n",
    "\n",
    "# (data + 1) / 2是为了还原被归一化的数据\n",
    "show((data + 1) / 2).resize((100, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataloader是一个可迭代的对象，它将dataset返回的每一条数据拼接成一个batch，并提供多线程加速优化和数据打乱等操作。当程序对dataset的所有数据遍历完一遍之后，相应的对Dataloader也完成了一次迭代。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       bird        deer         cat        bird\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA1zklEQVR4nO19SZNk13XeeWPOQ81d\nPXejAbCBJgUIpihRFCmJojVYGztohxVhhzZeSCF74a218sI/wF44wgs5HN7KlmTNIiVSojmAAAiA\nIIZuAN2o7uqu7ppzznyzF/f7TmZXZUn2Thm+Z1O3MvPdd+999713vjN8R8SKFStWrFixYsWKFStW\nrFixYsWKFStWrFixYsWKFSv/f4lz+qP//O//mWmMRhl+5GQi4gSB+TfO8UvXRcvz8Uma4JAsLkyj\n0H7Zcj2cNCsKEXFc/Ou5Lj7P2C2/chw08mLaH3or8JXvemiYcfKXnqtT5djS9EQnDn80cXGUw8Mr\nAb76rX/3P2YP+Y3/8B9xiNPA/HIcUi5hOSb85Fpy2zQaaxumcbf+OYw/80Qk59gKj7NwcGyNI4n8\nCg4Z7aC3/kemUVq+aBojr20awyIQkVEeYhHYf16gk9L42DTGTz4wjermBcyNy+537mNUHZyx1+mI\nSP3Kj+GQ5XXTqHTvmMZv/9vflqflX/7qZ7Esvs/BYEZhEIiIy93j8yr81Q/Q21u3H+JEISbyhR9/\nzjR+9qc/jUUo40Qhf5MVnohMogFmGmLfFtxXDhtpgp3guCXTiHx09wdf/45pvMrBOK4nIgUvR1Gg\nk5///Cto/NQtDCUe4i9PNJlMTGMU44z/6b9/Tf5W0T2voqd2eacUp28HXrt6oyYilQrmFUcRBtAd\nz05HRNKCe+/UGE6fkWPTf3m/OPpQwCclPi42l9H4yhevmcbnP/uCaQRB3TQ63bGIfPAR9vPtj7Hf\n/vSvH56c4KlBWrFixcrfU/FPf/TN70IduL3dN40ij0QkLhLzL5UnOX8R+sLGxrJpfHxnyzT2Hx+Z\nhj7yPQ8P2pAKyDiJRKRWp+LAV4q+NprVKlp82eR8CXtl9Ob6eFGMRnitnTt3TkTynI98J+V5cex4\nzJcMzxgEPs/DV4qH1/V6JZR5stTAu2uU6ksGnWQZFcMUJ6oHJU4AjVAwbMfoHfq6Y/8ONayEDepe\nUuLbrF2GOhAFWKhE0H8hgYh4Pk+XYSR5jtdpM6Qa5eGclRwv4TBAt0mKT6IYh9dCX0R86jJ5hi2R\nRGM5Q3JqGUmBE/lc7SAIZebtrVM+/a6PYpzo1R9C30ly9PbFzz5vGst1nmgSiYjHbaa9OrqNeN19\nbp6UP9p+uGcaH334GGPRfeTkIlJwcLq/RmOsUpZSjU1ijEQ/4bL/vygIqsrpAE4CDpVc962Pry5c\n3BCR525cMf8+9+wN09ii8nL7Q6gzh8dd0+gPRqYxHo/+zjOeGisHwEUOfMzV9/W+RqNWh2K1vLJm\nGs1lR0QSrnW3P2HHVsOyYsXKwop9YFmxYmVhZA4kjH0o28fOU9Zxn0ZTl4hgyEOOqbEPqP9nBEF+\nCQ1HLYUh+qkt1USkXKUZmL883D8wjT67rdQAeUpLUCYTIk2hwtnpAoQ23UJE4hTHphm08dBD/24Z\nnXg09/bHMNBOJlCGa3WcMRgnMk/OE4BE1P+1EVeWMFMPn4SJwjl8VfEBaU0vDrFJSI+EKvkZ9eSS\nmjMFGMQjSE+pbDslLKbBlurHCGglzekxCQXHEv+Jz255/SXmarSbWLGD3lBEcr7nHELCIp2/SiKS\n0A/jhxwM94bneDKDJjyC3lKZMJw70CVk7o8x/u+++SG65Tv3J25dRf95JiJ1r2n+HQ0jdnYK1xAJ\nDnnt3vvgnmkcdbG7A5e+Aj+XGY9TnuHYyQTHTmjYDgh+C+69nOvjOnPuuDNER6s2hzMBmu7kC1cu\nm4YxjKytwFbzqRvXTeP8xnnTeP/2e6aRcmxZrlD85KlnYOnTQ5wOibYaGm2CEJesWivzRFiNfr9n\nGksrLdOoN5ZE5NKlTfPv8VGHZ3jnxEythmXFipWFEfvAsmLFysLIHAW13oQuXVtnLFK9JSJVupzG\nA2jLRQbV92APvhU/gIq4fnHFNBQJlqrQDFuttmmEXiAitWrN/KuxKoMOAJqkREMVALTlNlTclJ6y\n4RA+jlvPIULHxOMcHABXLrURJ5XSBTca4kRRDB318YMnpnHxEgKa4j7mmJXnewkbZcxruYYfRDnD\nr8qraOyhk2gfqKFCtLjURkOcTERirmRKJTwhmsh4xiCHJ27cx9RaPv16JYxBQ5nKjsiM20UhoULp\njGgu5G9yugJTYs+Uo/JbuEb5JBORMKTz0dc4HTlLkgidNHmhHQdgARiEuCyj420m6uck9lHoERHI\nvP7OXfRP6HHj4qaIxMfAs+pHPh1nNCDm/egTeKPevfPANDR8r1zS0WYyE22n0u0C4CQJUbZPf2WC\nC5Tyq8I7PQozL4JfTjCk5SRVMwsb0/ArNspl3CDr5wAJg7AkIr0+BvBgG87Bow5Ge+XqVQwy38ap\nPWzXPm+uKFJv3VNoccax68x8LCJS5lbUaET1Ema8nZ88we3mEzaubZREpN3CrXrpwjk5Q6yGZcWK\nlYUR+8CyYsXKwsgcSFjkeIq1mnTwhY6IOPzcoY7nFPwk1yhE6HgBozRLdPA9Q8jWJCQsSUlmdOCH\nD6GWP/epm6axs4PgvZxOtpV1RJoNJoCNaUbfVgWIwOidyxsIam0uAYkc9w5N45A+iGYN4Nflg/vJ\nPZyxsYyjOr35bp2sAY+GOoByHwqtW4HvI3qAExXMfqhU8BsNnjSOpJCwRRgPWXBIAUNM/ZyuTGJD\nh8GfQYDfuByMl+ciEjBYN2csqxtilfwEF7c/QLeByx9T4ddg0CigX9j1RCSkzy5LRzxWzhLXPwlt\nNH/L5MrQCycBvc917hlN21J3m6cOSo6hRzfud9782DSOOxMRaVfQb7lKUwZtDhMitb0D5Cft7MGw\nkBD6rG/CMVovw/Ha6w5EJO5hygrQhkMNylVIyxnpZeUGSE4hSnyviWsEWTVCqohIajSm+5YoXhPI\napxjmYaXJPNF5OMt2GrufgKo227BFrG+jrSqL3wBwaVRBMPFJ1ufmMb9+zhqQmyYZZmI5Exum15K\n3sUBAb6GNwuvnT4c4thl/0/YvysiV68gd2e53ZIzxGpYVqxYWRiZoz6YVGcRyZhvsdZaEZFkhGd/\nuQo1YTLC26bKl/PG6rr2gt8wwsWl8bV/wKyd9rLQOigiS6t49quh9dEeLIVemQFTEbKFElrQyzW8\nADWKxxjv6wz/V6PvZID36v4jGK3zJbwNVlpwETz4GO+WrMCPS+WazJNeQA2Lv3R8NSpDcdNchIy6\nw4CqSkqbdz6ZiAh/KJnkJ4519BrRqF9S7wf1qVKCF2C8g3TliRQiUj//KRxK83lBLSBlQNCIr22d\nZyoYZEzr+DDHezssVUXEnWYR4+KWK/NXSWYSbjTnPOJkjTbh60z5eZ2heRXmLcdjnkh1OVrFE77I\nO32oiu9+eFdE2kze8qjmDKlBaGibRgaV6dVZaWGmq6u4iA6XPfR8EckYKHd4zHxmxmEN6cxplVSx\n4iJoJtYZ7olp2tA0n5kp4ow0i131TpwMi9I4tZRzjJOxiJRLWIRPPY+d4PKa6v0y4l28dX+Lp8YG\n2NyE8qUOMSMJ/TO7e7scEp1DfHrkvKwJtdZJxPwk5r11+lBs948+FpFSCWverJ25nayGZcWKlYUR\n+8CyYsXKwsgcSNhqAUwtEXHE/YGIVAsgwY1V2LOLOuN0qJHeuAqz2T4jszTppN6CvtdjLvijx09k\nxvysNsujI5hCY9qDl9uIyxh28FWVkUEe0VZEzbxWq4nIcIigkk4Xh1SIPVt1jOTR1iPTWGkicqrK\n39RonG6doZ0Oc9JgufwBnRIJYKuEjPMqCBEOmb0hKX8UjWQmykWTafQTxS+TlOCa3QYBoFPa65hG\n98PXcHgtEJHmKvIwRjRzxgSwSapAA51kXO2M2LY3AhDwJryIQSgiZSbxKPTJ8vmuCZkJwxEmrCii\nyT1/5uMpHUWLAHOJjpQnhIQh3QvtFkCcZvYUOX7jObmIjHhIPFbrhP4SDTWKNxn01K5xk9DmHUdY\nqGopEJGNNWye3MHGOz7Epez1yBfCDab4yyeSTTSs7mlxiXB1bBqVVmUkYJ6htyjRvDR8VWIc1v4u\nvEbL7bqI/Na/+dfm35/90pc4yAEbCMj6/ve/bxr37sEesr9/yPEDUU4mwIAGSGqMWKWGZ4Va5YNQ\nU8poAeD17fbRyWBEED3Gctzd2hWRShVRljdvXJYzxGpYVqxYWRixDywrVqwsjMxLzanT27UPpXHw\n+EhEblxH2sq5ets09GkXMlm8yoCsi22ALJ9MqS6h3xqdjOfaLREZjqAo9pjx016FoqtOqAnx3WCM\nxvEePH03GN4VU5ceDgYyk63iEo8d70IZHnQn/CVOHbjQ6s9dwLBrSwBKLcKuE7LfgYNDk901a8Tl\nSDYLjJYZCHJM2JUw5cbMURFBTCeRJj1EHkOrCMRCIT4i4sgZqnNxE2i9UYtEJO+D0iAOANUPi7Zp\nVIk0C1670aiD1ajBZxrHdICqDyuLRKQs2BiaZ5KcwWlhDuZf3S8aS/VUnodHBos6fakNuoB3j7Da\nzTog2/l1wIeqsgDTdXXcORaRlABwzMVJ1LnJYZcYAxbT1xjRkzUaYJOEJdwOpcAVkYLbqUzyCY8B\nR+rdVj5IXnZJuSU0B+uEzNAe4BNdFo3lqlV4KzH+rrEEp/zaOmwmGUPMfus3/5WI/No//zXzb0Fu\nic3NkzrKtWvYG++8A2qEra1vmIaSFCoNZ78/kZmUmgYzzEJybcZDxAAIz5ikOLbXhS1IAyF10xx0\nJiLy7gfgFORE54jVsKxYsbIwYh9YVqxYWRiZAwk16TwZQj1+duOqiFwkU0KFGrvWKZlmb7OTKlVE\n1XW1W4e6YstzhJwNIrK6hlDMgtCgRX6vb7/+PdOYdEguXsdXHt1u+8zjyaQQkdUNaMud/Y5p7G7t\no38X81o/h0SflQ2AoMYS3FIT0rFHzny/zoSRnymdd5qd4Cr5PaNAHcZKpikjIadxoYHM5CepKJoo\nqFEXBKeOy7UlANfyQjXq6p+9sS4iEWMp32JpnCREKGAq6KTHToYkp1/lWDRhKKADL+p1RKTcIklA\nBQs4Sc6M9EtYJ6bEIEYNkzXuwoIxh/kQo/XLMBpUysw4oXNwfRkOuOUGViMkJOz3NDjWEZkBV/yr\nuUGaI5LnilboD6XPccxwyuUVb/aoYkIsz+vuMe3GI9RUqvhxhE40DNg9gxxdPy7R81qim1Ip8TQC\nVp13K6tt06jRuflLX/lHpvHVr/5TESnoqtNMJqauTZ3yzSZW+8qVq6aR8ooQTMvqCkwlqRyLSESW\niyLHVtdkmr0xrCuZEtuz0RliMPtHJEesVDgYX0QOD2FqePjosZwhVsOyYsXKwoh9YFmxYmVhZB6n\nO7PPK3RCra6siEzVVlUmNR1JGcX0k4DOwZlyWycZqdMkFpGIsZQeuR8C9Y8QGK23AD2evYiaTiss\n4TlKgCPOL8NRYhwx9z5kFNwukGCJCYlrm+it3iaQ0XpfzLDrDhmGykpEJyRTum5OUOeV0W0UMqwu\nmzBUknBYIaCJFD1VEFPrUUqFbqmcjRaJ4derdMRQM7/QBkhfq7oyU6O0IFPFGrM+7wzbpnHA8Q+J\nPVeICArC3ljjBpOJiKw0iARZqLU/PD45AUpGpBkVWFLdG2FYFpFE45Pp4aqwZKzPNLolUrttrtGN\nW2d6KT3IB4w3HgxjmclcSwhsNAMvyfRycF68dj0mzMbkt0iLjmnUq2WZLQ1HGO4SRe7tYac9cwmD\nLGjuUCTon7rQkELvKS1BRguDBtRSanSehvRUvvLKZ0zj13/9X5iGofTTMNQnTMuNY6C5S5dgHOgP\n4IF9sL3F3+BKqXPz0SOEWBsmQodIc9jvYEishtdstk2jy1zLmHwqAxpGHu4C+lUrzKhNCxHx6dnP\nzgivFathWbFiZYFkXkYFE74bLNttiiNqVZgSGYL6fRjY1C6oWeOqgGgtHP0kZT/GNlllan7GwYwZ\nmeXxnf/CNeSaV1jwRrNd1ldgL1+hT+Cgcywi25/gneDxlxcvohR7xPqgB/t4JdYaeIEz8Et6h1Dc\nHg12ZJ4UEeugaDlvrqXDiqTnqnw3Ul8a5dREmLDuFY7M07C8qWYKNTB2cDmaLkZ5qYZTJizLfomv\n3AxFamGivnYeI9nfxi+jPlPnqfi4JMDI9AUbM0efdmWjZVSYt7S7xzft2WFYysahgTkZG8YUnU0X\nkJo1E1naTcz98S7C7qasANS+j5h6pXVuosSVmairhCwROc3vzilGBHdKLkJqAVaxn0zQ6BwPRaRC\nwiktWdpsQo3a4XbaPb5kGpUavhoPcadosdITcnJAM6hFrfSOVhWis6vKwXzlF37BNM5tbM72NGIa\n3O/8zu+Yhh9gWX7jN37TNDQ155vf/CvTSCYYbUjAoSQfxkPihiefG5oJ95nPQNc7OIQZ/uF98JQp\nDjjqYF8dksY6L1wRWSZpSrPRlDPEalhWrFhZGLEPLCtWrCyMzIGEQyZVxyM8zgyhWpnhUdOsF1rH\nU2rdCgkDxmqpPVtrPQZkZcvSiYiUGF0yIDdANGFBShb5bDCqy2MxVAUaPtMUmg2gxc2N8yISE7K9\n8967ptFhvZABNd6AZvgmVfeIaLRE4/SI6TsnJGd4kaPBVsS8rQIW7hUflBVlGl/HxKduQTY+ryKz\ndVB0uXhlxi4G0C8YsOZ2TGODKQxBABC3Ql39cb8rIh9sIU9iEGEB393DIOMy1jYbgabWi2B8DZcA\nKzKawyv8cTF0RaTTgf4/jAj3CFJOy5Q4WNScrA6ZQmbM2Oqx0TimFiFhifFlutO6fUz5gHkePab+\nZ1koMgVXamsvZtAVP+FXDKHymFUzYcXUgJfMMEEOxrp5OMgGLy7r/r53Bw4fLSY6HuGoRmN+wJo7\nLUR6iuqP66Nhjx5Zp9vMjHnmxg3OHetjaI7feust8+8f//Efm8Yrr7xkGn/x539pGr/7u79nGhF5\nFTT8rcaAr4x3U1Yti8jGZWBepZTY3MSeuXXrFmbk/ZhpvPkWXBBPdh6caESkCDU3ka55pVSVM8Rq\nWFasWFkYsQ8sK1asLIzMgYS9ARPWGReT5RMRGY2YBeMyxYHk4h4B1NRV5j6l9ouIFxA+TE9kPFbM\nbCBukoSKokagEEB5RJou8zz6x/AfKc9ZGpgKHOAAu/MRGAv2ngCgbVwBrV2JCR+TI2jsNWrdh8dA\nPZXSfB1ei4yWCng62k7HNC5n9zAkFyC0QW/dZ0J8UnB9CieSGUioKHtIXocxkc4KXTZrDEFy+lDU\nW0xUKlXR6HQzEdkmy+BewLC1dSjbFV6pVoSZ7j3aQqN4l2MjxwBDmQrPrAYG77v08qRnxmF5Gk/E\nt6PGqZUM/uJKanqKUq0nTH9pNWgT4I+e7MAlNx4RuymdYZ4LHU8yw6KhWSlTckQ2fIJERcEev2yR\nKTCJXBE5OFaDiZpB0EmJM733iOyV/G5zDbtoY1PD+u7JrJwiaVDHpeJYn0hQd8u5cwBimgn34YfY\n7V//y6+LyOuvvW7+3Se7yZMnsBL8/u/9CQ65s2Ua1+mLPzzEjwe7MG4oY99zL39GRDYunefYSIXI\nG2c0wnZq0uV35RpiJ2/eBFp8/903TePb3/7W7GQHNEaN2clpsRqWFStWFkbsA8uKFSsLI3MgYaDe\nwKc5GNTNp6nnMd0i7RYhSQnupGkGA3VF1cxVaVxaWpYZzNjvAy5NGd14Ru02IXxguL80mP7isgTp\neDgWkWuXQTf4EhXRnR24w+Ix3JHjHkNhOQZXS1HRU5m4J5EsfkkkWBE419oJCsG2ciCCFWYLNZok\n+VaFn4ts+NnVlaZ0gAHXNmBls2ktsAHBQg3Loj7TfYKpw2xFRJwSFsejh7FEFnDfwSGN89DYl3o4\n0c59uJZ0VJevYFlWV5ZlJtY0ZZ5H6M/ZSPiqpBuAqV0cpKkvqwQJlSpGG5H5PGUIa7OOmR4eA9Hs\n7gKElgiHQ7qWRpMjESmTCSDNFF8wFveU6CZUXormEnqrVbF0e8ORTKNfp0gtVn8iUfaUcp0WjDVy\nh2go7AlRXOkyR8eflvliAhw/6Q+x7BPy1r/7w/dN44dvAm093n4gInsEpxGTw+7cBklei/WMNUJZ\ne0tZuNjhzXX1GvJ4nrv1aRGpV7XkLasXK8EDUfyojwvUYOi4OihrZeboMBnok7tbIkIOCDnucM+f\nEqthWbFiZWFkzouxROu1chYba1+bzLnFNI2ZxTNYbbQgwdDpZBNlWdXcV/PK1UxLJZZV46I35S1i\n5Rhm4cb8pLKBjJw6DdvD/X0RSdjtM2T5WWrgVfzxNuyd59YRIVKnvjMew9TdIEmQZvackOUqBlCn\nBlSfQINrVpQuqsK5nzQAB8xxDT1DE8w4I76dgwLvpRLZtbQYqsa4rZ0jMa6PwdzZx48f5+siUuEh\nlwIMsssBTDyE8PQD5hU/9wrmRr6wj9//gWmMO7BwV5dbMsOIXa9hSbvD+RknMsOzrMFcIcN84iiS\nGYOxqjkRaaRUu1H7+A49JxPNXKFmXWH+0zI4szDTXv9MC+5UuG0rVexkVUC61OnAt8V9Pc1S5i7V\nHa7eg5C/aTD/rBzOV9inqc7TnOqTFFpKVK3p4n3ynb36KtJrOkwPenj/gYgkY/rNyAGtwVzXrz9j\nGrc/gJ3+8WNkoRVcOp+29ozul52dhzLD2ry8hA2wxlvpylU4u/RRoI33fvQ2zngH+uBXv/pPTOMP\n/tefiMjOPZQB3u+cuZ2shmXFipWFEfvAsmLFysLIHEioMRdqeDbGb82o0IiqQOOwvJOGVYf8Dadp\nGxQSjkZjmUFDIbVlauUSEGnmzkkzfEhioe+99YZpPGS8v6HsaVOlf+kzL5nGV778ZdPY/V2ozcKE\n/udvPmsa3R4shfssilkMyVjwtNRdfF4XqOV1EieFhcagof/BAIis2aR3gvlJBtFkNPZXmS3kEhIG\nmvqT4IxlVi2qt6CHv/MAo/3wCPju2PdFZDPdNv9+GvhPUvoQ7tMKe59ZO0UFx156Bgn308isR9DV\nX7xwWUTay1T7ibKPBw/lDMnJ9DClKiaiMZbaVMmmT5FDOKyrokVSC+6rnDttTCuBqM0hLIvIgBzH\nGtX1t4gSPx0dw/MzZtRhTPKSvHBkJvJIaamVykobIe+UKm+lGj0Pntrqn5bTRgO9cdS4oVWF1Gai\n5pSYdUxDJgMZ9mQ1z49Ybmp3dxeDJbbtdrCBlXwlUCSYYTBb9wAbDw9qIlJWdM95KTOXQ2Cec/w1\nMqyUK9hpN28i4OunPv9509i4cElE/tt/+a8Y5OEDOUOshmXFipWFEfvAsmLFysLIHEjo0EkUUyHs\nDhMRWW4rJGR+DHVUzciZ0gRrhRL6/hxm3tQY3OGX6yLSZ/iPGwAWhWT+1aIdCaM8qj5A4ru3UQbm\nz7/2F6ahfrdGpSoiuwdIL/hkB7DopRdfNI1//Ku/bBrf+DoS1n/0DtwWJTqwcpJALDXnF1JtEpMG\nTDQpqJ87JFGYkP1Z56gKf4W6tElPUQdZQt64MY91XV038rfVsYBPekBMHxxgWXoOciYm+VBEtncR\ndHOdS7m+SiaGLoOhojYaLKhTquCT8+cB/R4yl6LXqYrIiIkyQufXhK7A0zJiLRytjqOIydDCDUeY\nxTRiiyhYHbQky5Mm594lXV9K6DFgMlNWjEQkislhrWlP7CQ/6cGeJutMWNspZsOdvtQdmSFEVgDo\nMgfIJ1zVyKkmCfZq3FfOaapGM2beLwXdhZp/o19pDpPmvQ0Z0jhmCFOVkPDatSsisrKKwrof3cdd\noITIB7xBJiyCOxMoh6s5YTCggvUk6csMnYNeU03Acng7jMaA6ilD10ohdstHzJb77ve+Yxrt5VUR\nWV+Gxz/yDuQMsRqWFStWFkbsA8uKFSsLI3MgoUtsdUTGux+994GI3LiGSLOQ8ZAOy+rkLDHi0gOY\nkb9ZNKyOX6nWXarURKTJZJFuF8rq1D/CAMK9Q/j1lI3v1TdewxioOZdJNG6qjJRqgC3K+f21vwB4\n/NIXfto0folM2H/2p183jfu70JzPXwcBfIO8gCekoqGeGRqjCSuvkHIvrGBGmo2kvSV0XYWlkoiE\nZV03MhkwYHVCd5VHx2LCSLxtEs8/KaD5j4mdnCQVkeEIh3xvB7za5zcwkrtMS4rrV03Db4NIICRk\napMm/4j+HVPLM446mGkGaDAc9eQMGZNWfFowlT6maqUiIpWqJmBhJ7Ami9CTOS0Z2y6zRA3dajkR\nU6xHxanMeMGmfjltnVW6ZkZmSNaVgFBEJCD2CUk87yr5HyGVwsYGY7BDjSk9AxKqQ7kg0tQir0Kk\nmRKgCWusHh2CTWGHVW1arC+bt5oicvPmp82/X/75L5nG91+DV317G47diOR8Khz1NEBUCS/yOBaR\nCTPjqmQxrCjfPGlUQkYOdFk/VSuqHuyhTur21hZn7YjI+eW2+fcXfuYqzvftpzktrIZlxYqVBRL7\nwLJixcrCyBxIOGAkniYQpVKIyITxfo2yxouSCp2lkBS21CvwWym+G08dSdB1o1xEJOLnit0O6b+4\nxwJBtz+6bRqP95BK5rNw4xqjKLXQUGc0EJEqB1/RYpakfHj3h+/gkOOOafwiY0q/9td/bRp9upyi\ndkvmyZgZailxS4sEcv0+YvPygkntDQyyWiZtA0c1ilMRcan/h9T2lVy8xFphfhO47NEuFqEb45OU\nOEVSotFcRCTxkf45KiEh7tEIajlZGiVlodM64eRIs9hIB7i5gcBTN0tFpEQXqjtAt9HxfTlD1E2s\nHtildnu2USMnukYsj/vYEvsTeqloYXC4Ccsc7XhCyEYCcsOkrlQiU3L3M8ka5ojCR4V+xvenjksN\nHJ0hqiDlBrGP+oJnokzPrKR65vdahFUtMJxJEuNyHxzhlqnVWWGs0ZIZBvoff/kF01hiQPU3vvFN\n07h9GzeXFuurKdEFqRemNQfMwnBI9RomuN7CsdUqrnJ3SHchI3h9D51srKP/kuB22OsNROR8E508\nd/XcqVWAWA3LihUrCyNzNKxUTYZULurtpojskTh17RwM0iWG4TgeQ7cYizRgFIZ+0u3BAq3pNZeN\nFZ/Z9sf8wf/+LqIzHj7BezsnKXN7tc2B422jBe419ae80hIRh7bYhmb8rEHd0II3DxmcwgwQ+cLP\n/IxpfON7f2Ma2RnvZZ+hZy6jlhoBTNTNGNrZHr0WExrm15k97zIFJ05TEYkZVbS8Cl0moq5UYjnb\nnSMce/8JLKwHOaKu1CjqU10yb+oqq6o4ZVyy0KOazISWUYD8Ho/v1YkwRox0WhfW0Bh3HopI7Vmc\nt3cIDavtnpleXyJNVbuFwbSZ0FMNSyLi0WPDTSQBX8Uer/uE5VVSVnstk4lJfSIOE8UycUTkkLTX\nIxIVFP9XVne1qeOScY9LteSJSMpKq9MQJE230WHzrqrWGK/EC0THwEmZcmRPqVCcE19N46QUtfAu\n0xC2tTUQb62sronIhOwjVSKSmzevmka3S564R7C+azaSsqRoyfgTep+6Dho0um+u4zrUOOUadfjj\njmZc4UK8fAvBgDevXzeNu3cfi8ije1vmX9VMT4vVsKxYsbIwYh9YVqxYWRiZAwnrFdYoFSic+3u7\nIrJD5XhjDSaxC5ehlpdq+OqYJVjuPdgyjZ1HgF37uwi+qNL4/ZPRUEQuXgCX8dZd5JEcE3tWlPas\njoZPw61mruRMOslSKsyTSESWyOdXYycDGvW1WupSo8mvgEYf3sewv/iTiNX65L23ZZ7EzFZwGW+S\noKKMVOqw/l5eAgj69pvo9ijFUattElGMxiJSZzDakNjnyjWkxfgkcri3A9X9yRMY9Y9CdDKuQUUf\nakaU54mIyyG55asYtnI0M6u+5GIR0gmzK0gHGDAgKyQ7c687lJkSLI9YumaKgk6JopVKGRcipBm+\nyHKZQWo5LdNKZaEWXO1eU0Aq/MpV4wCxs6nxMxiQCJvFn4rp61kRzkm8r8ZxzbNxyR1oCup4moWm\nNJNcSUWc6mcoa3idwrpivoowpTlxT/5A4ZgWkRWGNEa8C/bpjBoyEarX64pIPBly8GTErqPx6Ref\nM43tT3CHvvfundmZzo5KV8x8oqFnJc5UqZ8bTbKMlDDadh0XSL0gly7AePLKy9dM4/qlTRH5Vo7R\nTqIzLQxWw7JixcrCiH1gWbFiZWFkniZPzTMmp7sJt3+yA+fUqyMo2zeO4CS68fxN03jjTQT+v/Ej\nVF4JqVovtwiCchz++vf+RkSaP/cV86/SHpR4iE80N2WRp35cIjV1Hmh2ONTIqD8SkTwAQIio1U5d\nKr7GQEGjDgg5trbBHOYx0efll+BM+cvv3pEZ0RQKhqDJ4RDY4cIS0yNcTHkbiFP6BDv7XSxdevRI\nRFaYsvPJPj7vUn/+CaruzRYU6d23f2gavQZcQkOCuPFUhc9FxMtJEiCYjjq5HHqAlAghHQFNpFzt\ncgpHW7+DCeSFJyK1GlDkQQdu3H5xZoxTjYk4CvSUrM7kaijk0RozIRd/iZg6omN3lJBjgAlegRLn\nE1Z7cSYidZ53NOZ+1mQXymmvoU98FwZcBFohfKeQmXQ0dxqHpZCH24kMl0rqX0zXZz6Dn7rOp0Pi\nIVqkKtOKs/TiKTfh9jZg3Tq9hM16Q0SatKW0W7SQkE6vfQXb6Vd++edN4/IlmHq+9Z3vmsbR8cn6\nuMZlmeWsgEUP7IgosuZi2f0KiTmZejVmZ0mCTwJWKb5wY1VErj0GQnz48GRGjorVsKxYsbIwYh9Y\nVqxYWRiZAwlrJaiR5zfAATAc9GWGFnqaH//u26bxqRvgRF+lDnzzIlIEFHEUVJgjBrN1jjoisnUP\n+Tc//hKoxB8/hDvsPrn3Kg2ojumIeRh0T2iWQInPXhNBmpPSbECo2GYcbLkCrfWYGm+f0a03X/4x\n0+gx5vPgiI6wp2VCD9pYnR+aCuJA/X77Dv2SJei6WRUhl6M+wFSrnYtIawNBraTkk2++AZIzZYN4\n8RVk3m9euopTDwCCtATmUAkUnUREAlI+OA4jexmmq17CIMP6uJxRnDAWN8EihIrEay2Zyebpknuv\nryGGp0TZ7DQ9RaGTYaQLGKKpdd5qrKjqMt5yeZkn4kyVCjFntoxSt3tOKjM06mWlgWeop7oC6ewS\nBYvkv5sWDfAJYKuV0ux5NedMixK4dFhXmZd2unjXWSqCxmpOQ0wVRTJ02XFOwnm1kKgD8bgD/obU\nLDLpNGKmZAUb+ERvhxdewM176SLud59X5A//6E9wOPs3+DRWCvwBLuXOAR4OemNm9MBOWBZ3zMaD\nh7in9g5gALl87ZKIrBLPvvnWD+QMsRqWFStWFkbmFVKl8fL8Rts0Ro2miOzvQfFRcpyUQUMTZtW8\neA2x9ussRPrh+2Cw6vM3Zb5tTE0RDRVRW/vnP/cTpnF9H2pazMIqn7C2tab+qJWyRr2pVPdFpFHD\nW26fT/E+S9fELMqSMqnixX/wsmlEDJP5aOsT9HapLfMkZIX0uI+xtQJ8steBYfid+4iOkXOfNX87\nE5x6tI/VSOqFiJRoC0/ZyYiZRt96D74ObxORWReu4pW48zE0oJSlfXLqBbmfi0hCkmstLp9oWVaG\nEUUZjg3UyksL/YRRdU1qSY5htaY3ZjDGdP4WDStQ4ip15jBlt1oty0y+tyZXaf0kNTyXGNC0wfyq\nmaqizJUR/Ob4uCciAZm+K6p7K5k1u3XUFK4hYCXyOjHnWTl/s6wQkRJ7UzO5Wr41iatSrvJYXQbl\nOz4rOYfBaGxk3IrONAdI865YUpcp9I6De3Z/H3vvwYMHItJ45gZ6Y+hWp0etmcRVmjGjXpHP/yS2\n6717d03jB2+DL0BcV2Z8FL0httPDx9TTWVw5py7fG6ARxTm7RUjmnTsIvVxfacuMQ8B3GnKGWA3L\nihUrCyP2gWXFipWFkTmQMJ7AKL6/x8CJIpPZMh40LvrU+h7cB4CKI3wypKk7paKc0TZ5wOgtk3D/\n+MmO+bdKU+vmJizTVyuo46IlTDxGwdylshonZO9i1kuSeiISE8X4msfQ01wNaMUbtDKevwgyg9//\nQ1gZc4LQMmOOTshulznoMfmwVgAEHj/CjLaPofqurGOQToxlmdDwvLU3EZFj2iOrq/BafOHLv2oa\nT+4jJuUHHwCSP7NMzlmudpFonpDCH5GZGqU5S8hkRClRSmhDJJXkTGHJMdqEZWXLBfrfWFoTkV4f\nCxizxk92RniRiDjcAGPGefEaYrSai6JpNyoaA9VqYm/EEXpbZYWVWD0AXVa0rZVlJgdlQo+KT5Sn\nlVynwWiMnCoxeshjHJZDoz7s4sqDTMtIQYAW8XIopJ0yC6sVIpu/UHpzKdfVtMbPKReBQkOtFOUQ\niHWPscFu3/lIRM6tgYrDJ9StsjrycMgEtRbwV49VV7V004svIgjxR++DMytG4BUmFtHEcXyMRZas\nzxmxmtGYNGHcco/2YFJ47U043C5fuiAitQrsIRvrG1yY9+VpsRqWFStWFkbsA8uKFSsLI3MgYa4a\nuxbA8EsikjKwYjykjs1Qlyd7T0xDY/kPqZ+rCpwQP2qeTZrkInK4B4Q4SFAbdXsXbAQ9ZoSUqGNX\nWL7FJ1ub5590EnX6YxE5PkBAypi+iZj5BKpItyvI/LjzJk5dIifvJon0btxCZJb8z+/IjDwawJUm\nGWO4GF/2/LWrprEfQNl+QC+elNvmb3gJlLWT3lhE+up3cZlaUcNoj+g6GfcxkTDE1BosreoxPygj\nh0RmgAR9UqlGD+krStEi0VxBQucywXXdxSLXm1DRg3pVRBzSELaqdK6dQUIgIse9jmlojk6T+M6M\n3jmVIaOFajzi+gYzS+JE81SwN1x6A106puumfCn9ocxskQFjkWJSGCt5ZBBqfg9Xg2NQtOV5oYjo\nRFPipoTez4zRSRqrNVUINP/oDOpARYI5y44qRWVBCJ1OnZuMEVMXOUvbZkL+hoMjEdk97GDK7LZc\n1RwdHFLnPUVv5NTv+bnPfc40HpOf4+t/9Q0RcXQaHJtW3jnsMn+Io9VyyDr+MXH9+x+j2zfe+lBE\nXngWdbmWl+fbYcRqWFasWFkgsQ8sK1asLIzMgYRjYqshAUXsiIjUGcvvCJxEShQ9oWPRoSLdZomX\nLsM1fcYHarBeqd4SkYDK8IMd+NQ066VJXsCUwXt3H8Ed6Xua8MH4OjZSNxSRLmuXDgiXXI9JCQwx\n7T4+5GiBJj59DTk0KQnGl1faMk+UTcGltq9VhV56Gb6VaBmBo49fA+wdh/DaFGQrL7sTEXEIV0P6\nNJ2YlOTUzzMXq1FhMZsGwYh/RGxAt1dhcnH4A5eqe0CA5GmYaK5ohfk3KbxFFVE2bsYuOoGIZHRK\nNmvobYNhtKclYsGblo9Q3pCIKXBcmXHDTTnXFbZOERR+015ekqd/9PAB8rdyelqb9aaI+CXA2ILu\n6aMjzGscYfypFuPhdpoGcHJZXFcjIUVEJkSC+oOpX09tKXR35koqyQXMir/LS6jZQlMUqVrFyd40\n5rPKEgHKcLm7dywir/8AYdvPP/cp07hwDi74oyNYb+IJVmOJd4FG8DabgGY/97Oow/rq918XkW6/\ng5lyzjkvx4ROW00bKrg+8XSOWMMOHhvy0Ud7ItJknMASKTBPi9WwrFixsjBiH1hWrFhZGJkDCVtt\naIbbdLT1hyMR8ehFGpO2Ycj0wOWWIgKG1bFRorMjIjxR10+eTkQkp1qbs7DV8SGQVB4BRChaGTJk\nUbO0ymSI1yDGSr0pIo06jp1EHdPINMTUwxmf7AIjhD705HVWDD3qA8SNOMcTMkrhjgnplxnR7+Yw\n5q9ByOyTMnDikHaCEZ6NUklEylosMyAvHdnsxhovGFLHzggbWeeypNyHVLaLuC8iTqaDhCjTxoxb\nCvgiURIF8vY1G4RFzPf0JpGI+AXW7cpm2zQq9TOTv8p0DmoUqKelR592mbnTwlY6yOLEIS6xW5Ph\njsuMIH2wBfLFw6MjEYl5FRLiMmV/DwTg2uPCaOLb6aJuCbkD0yyf7URZBnUlI7rKNOZzGvBMYnjH\n10vxlOhMMx6sBdtIzjh1p2aaA0nKjeUWum00sRqTcSQij/ewew8Zrd2l03aFt7nmKjpcBP3K5Xa6\nsLZmGv/wi18UkT/6sz/FBHmsBgMUBLDKdM8dN62Yp6HCWkttOBqJyISxrCGdnqfFalhWrFhZGJmj\nYS3x6fbshRV+tiIi0YQZLW3WQaF2E2iKNq2BCUNdCiZeO3w1ZQyQMa/clHEa1zcZY0XDsz63Y767\niou0ufKJ3iS5sFbEdoOKiBQsXK51RLQWiGZdaK5GyJgUnyn49SW8UpzJfA1LtYCEUUt9FuOZMEyp\nRAfDUgWDOWJd9cjBS34orog4ygrtY5CJi0VOaQKvcVVG5OqKBxhbnW+zI1YGqoaRiDjJkLPA2Dxq\nMarnJuTzGo3oHsmxYpvnwZbhURcK8qGILLcweBqvxXMZ2XRKNIZO6Q3UnJwWmYiEoubtk9blmSrz\nTPDKcaKUykuLqtblyxjt1v0HIvJkB3q6moGHYzpKuF2VgUuoAU1N6dx7GpQUGl2YHhtFBmmigVoa\nYMhYJNUZ1YR/Rg7TTCl4WtY1NUcXX3OYqHRP+RuYVjXqIbLJbKgwxNj29hDbeP8RXFsetf7zq7jN\npyQZ3AkqzSbuu1/5xa/IjL727TdeM42Ed/E0qu4kndd0jh5/o7lYqGLPx0eRPq17z4jVsKxYsbIw\nYh9YVqxYWRiZAwlvXUPxjKvngBq2t49EJFiF7r26BhQWlqgDx2gcHgBN1Gqk06sofwBtusylQABO\nocFcWnQEvY0G0CY1I0dpmpeXMQbV4SPGWxmMGUUZu4Wde518hFrnUlluC8UetDJWmJbkOafMsEYI\nSXxa3wtqxQ8eQut2azjRS1cQftW7Dch2xByawA1ExNFgMdpcAwLY6gTqdz0gbKHnYaOJ/qsV5spw\nkZdWaiJSLkgGvQ9EEHBe7TZWY0wkOxoRuzlwoZw/hyveogfD5G0pQHMKxOmUAo1jOilKMaxYKNfM\nG9eXGaioNMHT+p3T8DLOnSZcZcJTaFZj1N6FCxdEJGLo1nGPQYIOGmPdKoluAO1WyRJwuO494z3Q\nGqIxoWLAS6bAudAsNC6UNuQMpsMpJJyGX2mi0kl8VGbU1WoLY2vWWGCY3CGm0GyNNaN0XncfYHMm\nxLzPPIOyTH3egLsP4b64eAHOqJiMF6aO0c2bz5t/X30bxbGEd4FMg62ULrF4+ptZRgq65owVSI89\nK4PJalhWrFhZILEPLCtWrCyMzCukWihkIzVa50hExjHUzvdvQ6usN6HRXbsOuvEPPiIBOQOOsgK6\nouNAq1Sm8ErZ/EarTqK3SyzxuH2fxOcxymwMWKXDcwFtNLpnOMSPL17akJnImkd0izQf0Y3Ip7SW\nVk0ZguSx9M2N68DFz95QLrGnhUqywzyVCt0ugyF6W29ibM+uoJFcwYLvTnC4myUiEtGBlToM5tqH\n2nylzII3Y8C6F+kOq6+yIikr66RXNk0j8D0RqbFqTvYcSCAyT/1WGt5D1ECMo7R/wsGETE9xpCIi\nIyIpLTvqE0qfFo+JOAr0NJ8jKIUiEpJK3HFOedDUW5gr2lJIpQVjuHT0zZlaNZsMHarXgZuWWjAj\nDMhE3lNOR+0kPQkJNd4KVX/o49YKwRkbWh+oUACrIU4c9IRlEE5OVJ2DCqAUPGpEE51qWh51g1aa\n5TbmeNDF3ovjnsz4grWk6/7+PsdPwwt5+xxe94icJd0u3IVj8nEeHOyKSK3GJWWmVP8x+Fq01o9u\nMAWAirt1jkqCWC1XRKTCel062tNiNSwrVqwsjNgHlhUrVhZG5kBCter75Ma7en1VRFIWZVRXXaUG\nrbXRhA5565bGmmqlJsIHas5RrNzh5gdUC0lHF5apdT9DarEUXymld06aQU8Z1+gRq9VNwUtMbW0N\nKMkjHZ0ij0IaPJbqfYRGnYMpivkOC49pMUtkkssZoeptQk8OCJT8rGMan7sGnDJmeo2XFzJTq2rI\n+q/5AAzuL56DhzHLcOzaEjNLSsyq0aQQlsPKs1REXKK8EpOEcl7w8YhXQeM2+fby6fYqiAQ1kM9E\nsVZJ3KjgOp/v+xKZSUvSt6NSIwRhSURc/sAhVFTijZloyuLEJ1l+MtRTIzyjOBaRUsjwZtI0Li/B\n+zkmy/jBES7ZUQ9bOo4V2ypdOjPJ8kxklmxQ3WFkuSDJQUIPcj4Fc0onPx8737r1In5APkh1eipk\nfvwQ7P5NxhA3qrwQzkm0FYaGs4QM7gOk2b1AB5/Wzd3+BOURBl1m2nGh9FbR/CoTYWso2EXklZdA\nb7mzizDd9BSJxdTDy7VT8n69RqvLSyKysY6tvqQDOCVWw7JixYoVK1asWLFixYoVK1asWLFixYoV\nK1asWLFixYoVK1as/D2V/wPor/K15rUhTQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7FFA9E5B9D68>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next() # 返回4张图片及标签\n",
    "print(' '.join('%11s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid((images+1)/2)).resize((400,100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   定义网络\n",
    "\n",
    "拷贝上面的LeNet网络，修改self.conv1第一个参数为3通道，因CIFAR-10是3通道彩图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5) \n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)  \n",
    "        self.fc1   = nn.Linear(16*5*5, 120)  \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) \n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  定义损失函数和优化器(loss和optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import optim\n",
    "criterion = nn.CrossEntropyLoss() # 交叉熵损失函数\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###   训练网络\n",
    "\n",
    "所有网络的训练流程都是类似的，不断地执行如下流程：\n",
    "\n",
    "- 输入数据\n",
    "- 前向传播+反向传播\n",
    "- 更新参数\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  2000] loss: 2.251\n",
      "[1,  4000] loss: 1.955\n",
      "[1,  6000] loss: 1.722\n",
      "[1,  8000] loss: 1.590\n",
      "[1, 10000] loss: 1.518\n",
      "[1, 12000] loss: 1.482\n",
      "[2,  2000] loss: 1.416\n",
      "[2,  4000] loss: 1.385\n",
      "[2,  6000] loss: 1.365\n",
      "[2,  8000] loss: 1.341\n",
      "[2, 10000] loss: 1.328\n",
      "[2, 12000] loss: 1.307\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "t.set_num_threads(8)\n",
    "for epoch in range(2):  \n",
    "    \n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        \n",
    "        # 输入数据\n",
    "        inputs, labels = data\n",
    "        \n",
    "        # 梯度清零\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # forward + backward \n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()   \n",
    "        \n",
    "        # 更新参数 \n",
    "        optimizer.step()\n",
    "        \n",
    "        # 打印log信息\n",
    "        # loss 是一个scalar,需要使用loss.item()来获取数值，不能使用loss[0]\n",
    "        running_loss += loss.item()\n",
    "        if i % 2000 == 1999: # 每2000个batch打印一下训练状态\n",
    "            print('[%d, %5d] loss: %.3f' \\\n",
    "                  % (epoch+1, i+1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处仅训练了2个epoch（遍历完一遍数据集称为一个epoch），来看看网络有没有效果。将测试图片输入到网络中，计算它的label，然后与实际的label进行比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "实际的label:       cat     ship     ship    plane\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA0bklEQVR4nO19WZMc6XXdqcysvbq6\nem/0ABgAg2UwxAyHo5mhRIkSJdohWrbssOWwFXaEIxzhFz/4wb9DP8ARDlNW2H6wZYclh+RNFmmS\nokmKnN2zYkCgATS6G71UV1fXmpWLH/KcW9Xd1SNRCke45e8+ALezsjK//PKrzHvuci7gxIkTJ06c\nOHHixIkTJ06cOHHixIkTJ06cOHHixImT/78kd3rTb/3TV/RZkimFIACQ87zszzAcZkqUjLhDoZAp\nccKvpEnKg3hxpujbSKOqjh8DyBcG2Z8+An0l1dGiTBlFPGyS2IADjYFbhlJy3DPR0XIaNkcbRzqR\nLtADBxnqWx2eGb2QH/3Gv7uPCdnf3+cAIu6ay02ZzJ9UfrKDpCeV8QYv+5MbvNQ7uUdO8yMlhU0g\nd05T2/tPGKTtuby8fOKj3/rWOrWYE7W/u50pw8EAwLXnrmd/NmbrmZL3OYBC3qdiW7SMgpwWSdTP\nlFo1r6/nAAQ+B+l7PMjBQTNTZmZmuGc+r6NxH1stURJmiq1b/pnj371uj98NuJxKpVKmhCG/G+mX\nUi6VdXyeqDFTmjzs13/zn/EqFm/yKz5/U/WZWqYcDbkUu+19jU2/C93XQMMtB0UAJT/QuHUr7dZp\nQ5zEJ7Yk2jI+rK7R83xMWwC5nP3e7acan9qH3yoWi5lS8Io6dRFArsDJ6e1/lCk//8u/duIgHpw4\nceLknEhwelNoL9iE7y4kCYAiaBl54IMwCE5aTzJZkAu4aWhvG71bgoQfZU9/7Yic7DVEQ51IT/rE\n19j4XooDPpvDUB9Fno4TA8jJOisV9E7WdXmBvZx1RnDnFPZa4Nsg8KY/033fn7r9zyl/NjMtp7fZ\n2CLycgASe5+mGm0qM0qvXDMzJ779Z7ewTkutwjvlpVxswy63JGEPQKnAo1XL3CHQ4W0BFLVKyrqb\nnoY9jG0fro1C3gMnAACCQGaa7DUvd/Lai4IIsuTQ7Y10IkoGI1Itfk8nyMv6MHttNBzqQjRs2RQ4\n4/4mKQcf+XM8SJ4/t9inheXlZWH1O5mSxl2dmscZptxn5CUABpo3/VwQjghoPC3gfo8/c1vSdiEG\nSjyPSpqEADwzeDVvUaQVaE+AnD0lOD9zc7y0YnlGh+WNSLwUQK7I88adGs4QZ2E5ceLk3Ih7YDlx\n4uTcyBRImApMAcPJLTnhskQoz68Ihcm0Nh+fudwKBZp5UZLXR/7kPmZM5uSn9/QYzXk0OFOPBnM/\noWm7vUcbtRPyW53OSJcUA6iVBAQ0ttkKHZ/lEi8w8eRYFXTyZfDndSFhMh3sGAj6DDT0Z5A/zdHG\niMx2Hhvi9kl2IQLmI15yYOAh1i3LnT5jcmrLnyCfMewgx1Mbviv4PH7eiwEUPYF32y5/+bBPx7bv\n876XAt7E0VDQxuB8xC1pLgAQC+EW8vyKIUEIN1l4IZY7otfjGfd3dzNlZZFAJvPH+wWuDF/HtwnM\n6+0fCC0OFWewCMBoZD+uY+Kl3B5rbLGCIXGO11Wa4akXnl3htw4PMqXWI0gMB/zNxrUSgGS2kf05\nI9xtJ/IshjYMNQk8Y6nE2R5PmO5vtvZsBdpBIl1XYmtHS7EQcMmVy4o8wCA5pz1BDCAx++lsx4iz\nsJw4cXJuxD2wnDhxcm5kCiQMElqVlrvhJSNMmO4TURyFbE6F0iJDUhZMUSrK6sqtTGm39gDs7dMI\nzwcFnU4RQCVM9VHJlA/XaaijtMB9PH4U1ggbm+0mgI2nrezPWpEHSba55fIqT7RQM6BhKWC8xoIs\n2/hUOgn3lNX650y/+nMhSp05NnyqZLQoSQCMBLc/vc8kspVV5kkZrl+aJ+QpKUaT/ORD+oxJKCgR\nL4nkSRAQyHsJgLz+9GKuhEJekMSPtWcoRXczJziv5RoNFC70qwAGusCKXAG+BQ4Nt+hKuwPCrjff\nfCtTRkKjc/XXeNiiB0DYDjmD4VohniGd1KLbgqsWxk2mQ8IICpyBizMRQB4qwutLqSrmV6/olr31\no0wJd4kNL7x4C0Bulz+KYY4xx5ou4KjPCGNJwy6mPJq3oLikooQWPB1WSgCCkeDwSEercraLh4eZ\nElx6IVN6jVkOUpg91o0oJbzYXJoC8GLFauMzDSlnYTlx4uTciHtgOXHi5NzIFEhoSCMXNKh4OQhl\nAPCEm0IZ+YUCbdR4nD92MgOzoCDKF//SX86UN7/3fQCbB3vZn10BwCiiRfpwYydT7j95kimluQuZ\ncnH1CrcUWM8RClEWassAogFt472dzUypNOYzZaPzNFMGuqKVGo3hiooD4pCI4Kwn+uko4f+l0pzP\nxIwKbuZVGqW80H5nCKB1SLP/6R6rUsozhAYLKk+xWhMLmVmxztTxjc/6p5OCHAipLi1vZSLxEICv\nMF8uJrjLK1Y7MhAhqOvXDUQo6Vc1NElkML4IoNNuZX/VKoRFnmbSKmYCBYNbCg4221TKSrkMheHC\nUQIgKNh9VxQv5kgi/Rysdq0gV0OqlZbE0z0M4x+dRfFSqyRT0qeAWU7YbZBTjVGscrclQv7eUQhg\ndP8Tjk2ek0QVQd1AF6axFSKVFj1WmnGo+i0Fjge1EgB/wD8DXjGGFzik/pbqn3JL/O7sIi9EJxp5\nFlflVSdpAsCX9yDwzlzzzsJy4sTJuZEpFtbQ42P7sMenchwNAczV+EiuqyInkGfd/KnjysrkZLpH\nr8uckW/+/n/KlKcHAwBPO9zh4SZ3WN98zOOXaGpFPs2oWp1P66BS0z58LZT08C15FQB7MpHWLl7O\nlIFsrvv3aWE1W8rlWePRri5TyQd6t5zhJbXKjDT5CQyOdDxBx7aPE1tOWVixJtXKuX29aa1yYne/\nnSntLq+oP4wBdHsqciryVnb7vFO1iswNjaQwHsyfcBU/kS1ZzFliEWcyr/rYLJdqnEiV6HbkVKPj\nncxj8nOqEZE5ZlNppfgxRgA6R5yTR5axJaPJjKNLdU6LZV298+57mfL5O5/LlMSSwuIQQCm1dELO\nZL8nnKE1E42UPhbw+CNVyA+HPUyTWJZXovy41IwJ/cpCy9XSiWaPNBvLzMwqLz/LMaSH2Zi4w+Iq\nR5tXPfM2K6ihipyuwmLpCiNaeVXRDQSYqjNVAOERr2KoyQnK8pdrBQYLtPVyeZmiKU3FGS0fX4Zb\nlMsDyHlKEsSZdW/OwnLixMm5EffAcuLEybmRKZBwt0d7rBk1MuXb/+vbAG7foGPvl+4Ql81ZsbWl\nooiSwVNFjhU9yLeLBw+ZE9TsFQGkFfrCvZqyP+aPMqWkDI5QaTKh1dnMced6jaN6urWVKe1WE0Bd\nxnBJPtdHAoD5Ou3nna2HmVJ7yjNeqKt8x7MIgDE6HJNuz6gsZGPLtDYuMF88AaYYbZBhQy859s6w\nIiEDZh0hGvO+l+XKHageYkuQcOeASsbTMBLe6x0RDu/I+77xhNP1wo1rmfLclYscrdKIxv5+o9PK\nTfw7Ubrhne2I9+U4TwSUPDkQ+odtABBKSkUJ4Iu2oWCEazaBI4YRYsNWsT4ae/dDAN0uE4KePuWe\n1XpNJxI21EyGHe5TUvhot9XKlLfeJ0isFn0A169xugJB0WGPi6csFpBkyLURKw4QG9YZtDFVLKXO\nuKjGmYz6SLAxL5RdvPcpj/rGdzIlev2L+lYRQJoSkxYEHgfglda2eIG+mCSSqiqWUsVwRvzWzEKD\np36yDwAdLqf8Ct1HeEyAGWiSB7ucN1/em+QmM7MG4njwcubvzwEItFzTs6M+zsJy4sTJuRH3wHLi\nxMm5kWmlOY2rmdLb4+NsVFgC0BRU7IUEWfWCUmDGoTRDQ7RFBxFB1q5M+N22Ig6NBQBzy4zidRJa\ny4sqxLEIYKh8j4Gq0vuKAT2riEZPGHAnHAAIlJbVaurEGmRfBq1f4Imethmg3GoTvzy7KGx7hnXa\n6jNKVauI1zAwFGxsENrbwiKGBMdEesffGaeyura3mIM2P0/sXC7x0oYDXlqlyC2rS0TrGd9xt8fL\nqcoIDwdia9OFdcQ2F42LjRRaGqeA2UeTVzNJDoGzpGRsedrJIGExjQHUFGadNW48pY8VhY9KBo+E\nxD1d+5hmNxYXdjsEMFPl9jnN24MNUjPff0zlk3t/mCmtvVamdAYcWy98P1MCKLuq1wZw5yYpjP/G\nX/tapjyjFTgscbSDLscfdnmieqqkpP4RpkneV1mMJsHChYkcL8ZAWTvg8aMN5hjWC/ylHG3yjGFp\nFkAqwsvcFlMaq89wuYZ14S9wkZQ7Sh9rcZADVU1Fe3QgFAYhgKhNuF1sMnw/6guPlwmZWw8Y6y+U\nCQlnLjCC6SsXLFXi1RApgEgLL0zOxITOwnLixMm5EffAcuLEybmRKZDw1ksMNGz8gHn9tdklAK9/\nidsr3nqmhIqPGBrKiV8tRiNTZoT43nmPEY1agyb0M1fuAEhl0hcMYA5YrNNTZYBnTTs04A/fezdT\nZkvcUlGQsVapAdjcpm1sBRueQOK80gVbB7R4D5pU7m/R1l1bZqJdoFGdkKDOq4iF6UaKkEJBHFMs\n9GPVIYaP0uMppOPooRSrIzEOAMO2DdXZjFQ9DyGLSm0GE5Aw5xv/gTqXlHXLrE+M4rjjGM2pwWSQ\nP3/y88/ChI/X1zVIzuRRm8smHg0BPFHd1YHoIrod4v3lBaK5WpUowlfScmiUhAXx8+n+dgc9AAMb\ntLjkH21yXT3YYKi0F4quQ4FjVHh8IxW3aq3tR58A2BTm+qPvfDdTbt98LlOWGsRH/U6LI1F7m9Ft\nMpR0RLl3QorCdKnuIIw0RZDZk9JRlVvn1c9nSj34KV7REed25Gcs6UZEqQhjmSfqxqK7kCtgJI6E\nvFZyXwz6lsfZj2MAvQ7PUtXRBtqzqJ/h/AxZQGI9HDpaclDyankk/r9cbuJCMTp7OTkLy4kTJ+dG\n3APLiRMn50amQMLKLMHOs9cYEMmKpZ69yoaXiyNihtZ9Jl6OBFLiiGjr9V/4m5ly+dqrmXL1xfVM\nefNtorm52iqAzR0a6oHYvEoKaakmHB3l9R0e0MaeU9dMsx2tFnxxcRHAQFX2e8oAtFKyWk1RSGWH\nhgo53X+8kSnLcwQaNy4qNe64fP1f/Rse1hJHZfrOqEfm9auEw6+99ILOyK9bcmkWiUsNv8g+jzSl\nFuQqFIUajABDWY4Lc8pZtQ5shQIm2AKQl+muoraWAqMtMa4dHbYyZWQ5sQrwLShv8Mb1awDyVqFm\n3TknQOMJ+c73fpApRtVvRZG9QQfA+vYT7UCxWZpT5nBVgdGizpNXKmmgvEdPbb56gxBAoLasqeDw\nVlPE5wrfVmoNnVMEJB2r9eOZrAS1XqsD+OlXX8r+7B42tQNx96NHnNJ79+7xIwXPH+5zSvsKIJ6Q\napXrLdKVjmK7C0RzRpeSEwour3B+2urqunvI0eZ8H0CoZmUFC8C1uGck5F9Ujndba7JkHQ2MLlE+\njWFWnaq2DId9zZvwa0V1jjMXL2WKbx6GcWc53eBxPnIKjNdTcnbmqLOwnDhxcm5kioXlF+k2e7L9\nYaZ84dXXAFRnaZj4R8biIONCr9wfb9Ab93NzTOZChQUfM1VVPwQ8frlQwUQxhPmS19bo8P7wHot4\nCnJJtuVTvHqJ1t/N51lV32yqg0g9B2Bzm4knnt4SDfFhtcSUZDZXudLIlL7K0D99pPKgwvRn+kD+\n7LCv8nSZM51DXbq2xLef57dSI/YVL22hjAlTZUx2LFNrdp4pPGMiB+t3YvwNskmtACrhvzzausqh\nnuxwWpr7tFX7fdWRDPW2FKODUQtcvESf9OVLFwFUC7ZsLHRwpoX1zl2euqISDbvRg6gHoDHP3DG7\ny6GMmp2O5laXXCtx7UUyFT1rrarWSl5QBVDoqhvoiC78ZrN5Yth2a0MRRbR1RqsGu7zIZTM/fwET\nFT/7B5zJhQbP++rnuRQ3NmmntwecqI9UuXKaTJwXKD96eYYX2FHKYaBVGltClipaPC2nRMliOV+x\nCM/HhLN8JPKSstomGbwwW9V87bHm1nrwRCqJy5dzABKlvBnJnfE65CM1NrZMQ323FNsql+VpnNXI\nYeJ25M5eTs7CcuLEybkR98By4sTJuZEpkDBfordyMDBoMAKQVyFLpWquUDr/iqJbnQloQ/7WP/8X\nmfKrf/ef8LAqUygUzYaPAFy99kz2506TzteBHJ+rywQLRlw7VCuUa9cZAXjuOrHh4dvsd9I96gBo\ni/Q2iszFSyO/oXyZpEWre3ZO3Axy1fseL2RjcwfT5O/8rV/jkOSirp7qE1kWdDLO4XZbbAoigcgH\nJQCB8llS2ed9ZS2liXLQhCby8u4HZsznrdDnGKK0fJaBaA+MsWCu0ciUWCyAJZ/jb+0T9Ww8Wc+U\n64q3+F6ACdzqC6V+RmnOkSXCmatb2LDilQBcvMQ8pnDIkext8yt7TQZkVpfJBldapCu32drXUblz\nfY64tVScAzAQy0Yv4pyXKrrvEe/7uLerHPbm3Ih6HO3rP3UnU24+uwZgENJr/uDH/Mq9Tz7IlJ95\n7cVMuXSZS/rRe4xKmb88OaPopKBsr4LyChPR3ZUVMInEgHjUVutTEYSUZolbV6qKEaUJjrUsFQOi\nbBRf3oNxZOaUpCoPMkgY+ykmGBA9KQVDnzrsUOSLRtMS6NpjTfu49VQSYKJwzSgqT4uzsJw4cXJu\nxD2wnDhxcm5kijWYU3FAT9Bs0OsDyKu95dG+akRUiJMHQcSFBi3DTz9kKsrmBhX0iPgePl7PlC+s\nfhHAM88yJri2Q6V7jzvMFxqZUm8QG96//4AnWqPV3RLIGsl8fbqzDyAxqgRZvD3F9bxTDAxW1mMB\nrIJ4zsL9bUyTRMloYxtbH9UKrJgplzhjfdG29UacuvX7vMZCoQzg8lUWsj94zPr73/uv38iUSNGc\nko5WMUVAslEn2GnMEhF84QsvAVhaZHnEcxc5XV5OnIKy1C0SZGGj/jLxxdqFBpVn2Kwo45DrKbtn\njILPfvHlRcy/uMwxWOB1b+8xgE5XBAYqzbBksYaYyNeu3ciU+iyvqL5IkLin6HAi7JxVofTUKLSn\ncFsYKrNJJAQFY3kMeMsKYmpfXuWULs1RKeU9AEsCnnWlL+0/JO57+OP1TFlV3LO1/X0edp6jDc/A\nX4F4C3w1iC3pZ9jaYXCz2SFlwu4Wo5BzM0yZvPMC0ai1K874D0aKx1lU2parNSUwV0NuDPC5czwO\nR1o8L/vIvqtqm/F31VBHZ7QlZzvnlRmXt2BgCgCeEG58dlqfs7CcOHFybsQ9sJw4cXJuZJqBarUm\nCg1cWFzABBL55rtEeXMKAN2Yp7FXKljYhfhrd4cgLhm2MuXydVJ8+aUigEqdRv7iClNM91VC0VJw\n0PgBl5eZRRkIn1oJjpXvZ9FAK22xDMOBIoyRWMYXBSvgqQmryMxKinFEYhM8Ib/7e3/AsYn32lPy\nXk3h1BkhtSs3eGlLC8RHCxdYtTO/uAygJDaC1kfEF+9//ChT+sKvBibsvsyIrv765SuZ8qUvvsLj\nV2cAVH3V0MjEDjVdkdpk9awiRw1ByzpsoyG+/G02RtvbawIoq45kZZUTWKko+/eUNATnJwqhRMIH\nD0BT5HnttlIljfNbKO/RBgdQb/O79dkGdxYdXE/5rshFmKwvqfB2lCpWxGMAhzNZ0z6B2pdeWuC1\nG1tDt90CEAlgGp/9VcHVjz76cabcvPW8js/Z3lQqaUmFVifE4Jh1BkiE1I6ULL27S+/EQZNH++S9\nH2bKx+8Se16/ziKwK9dvA5hbFAuFQJaxSxpPv6Ev3+hGtC0Y9yI41mtuoh2sgo/a08LFpzsNm4yD\nj2POkuws9lOd3lsPzsJy4sTJOZJpeVh6WNZretPOlDHRUrQtsqC9Fp93i3UepypPZKQOKOvK5VmZ\nb2TKs3oJZJkyP3zzo+zPJ1v0ns7UaHPllYHywaePNDorPVG6hx7GnS7fvXML8wAi7bCtGp1anQMI\n5HSv6L1qRSEImfiT9DiY1eXpxc8/evt/Z0pZNEzDkJ71vJzKP/3Tr2fKwyeki92n2xR3PscyjkK5\nBKA3pHWQlxn7yiukOhqoGap5iG9cY9nT58SytLbIS6tXaPskgxDA4232B905EAf0Hrd0O/RJt1Qc\nHo7UKV4nsnJrq8EajSIAlQbn5A54FbOz02cJQHC8JhkTL8msXNnCI4FqtmxLocTDLi7R61+r8QJL\nCjgEGmSQ543IctBSFYJY36NZ5aB51u1JnFCB1bgMlZqnMus04rTE8RBAqNKTvi6nMsO0xIfb9I5/\ncJ/Wt9X3jFT2lLbPTHrKxEyVkvjBn5e9dv02oxa9I973D95i7uFbb9DC+s531jPlww/fB3Dr9svZ\nnzdu3c6UxlwjU2w5+f5Jw0qVXZNbtACSGBNZhCZWrBPLmE/GKWBnypgVLudjooouSs7M63MWlhMn\nTs6NuAeWEydOzo1MY2uQg+3C8gXt5AFIlLBz4SIhyY821zOlBTVrCegmbyzSLTdbp6Fu+ThXBAlr\nswsA/uVv/uvsz56O3+6z6qKn8hrzN682RJXV5Km7RTsRvaQffbwJYEe0BObKbXgcZL0haCAWpCCk\nMR/0mAY1XxGOKE03aXcfE6XOy8a+eJEe6Bc+z2qhvGDF++/8MccvO78mkqOdvS0A1TphxUKdO/z1\nr/08B6kcp9lZ7rO4wOybZpMT9eAh6acPW4Sl7cMjAEeKWhyIhqmpfieRQhAFebgLgvNGYlGvc/xW\nxzO3PAOgaFC6LGoBUVaclnmRTSehebh5oiTqAyiIZWF5ZS1Tcqo9KiiryMBpSZUrvgZptBbG/pzl\nBFmiWa+rQhxjgJI/PhU27B1yJp+scyabyhFqqKvrykIDQEl0EeYYTgOi+EClP3tqZnNxjUuupmtv\nD6a7k61kx1oRp55tkWNbmVmNBdYn/dxXuOSuX+dP8rvf+lam3F/fANB7WywUYih58SW6Gi5d4kEC\nRWbiyBi9rZBI12jO9DTFRD9gIxCx5k/GdTXuA2ttay29y+qTxk53D0CSnsSVp8VZWE6cODk34h5Y\nTpw4OTcyBRIa8W59jsZ8FAcAijJ9b4r590dvEFsdFljNn4Dm98pFHvmDDxm/+Nmv/MNM+b44c7vd\nNoCRAnM7WydDgZ1I8SNht4ZH7PZMmdjncJc2fOQ3MmV1pYEJa9YqcgbiQe70xMWsjqrRgGVDywFD\njWuiUR5GVs9xTJ7cZY1+W7GnX/3lf5wpX/vaVzPlD7/JaNGy+CGW1XW1rFSgUi4BsCI+3xkpJSVD\nRbLGDRZFSmPZ/oTDfrTDNKVQ7XOCUhXAzAyzfpYFZEbhyfhOXkjQSuRNmZlhkK5en9FHOQAdEfI+\nfcp7Z3N7WioCSpGnsJqSzhr1ZQDJmAaS96Vc4+lSq+oQbElSbTlFs5uaggRApBsXxRxbe19k3Hbt\ngoSdQwZPN9XCZ3Ve1U5Vpv5lPZwSQdFIh7Fw5DMCWbduMtPw5Reo3L3PMPHb732EaZITEvTEZeyJ\n+CTvW6GMsqIUxfMUGL1xk8TNiX4ym9v/AUBzj+A0UXHY0ycfZ8pzNxg3vP05fnd5RS4g/dKjkfia\nlcwYpzEm7ssUamzh7tMkfGOWx/HF2pdSYIwwxxU/p8RZWE6cODk34h5YTpw4OTcyBRJWa4Qtc4uM\ncWQ97weqXynVZC0rePToEYsGvvw62c4GHSVn1mlsb22wnuDTu3d52CjEuDEHOuJdqM8TilppTkMp\nrLdu8fg/fIeW7Vsfr/PUX/mVTMmIBu/f+/TEQSzXdKDqisurRHMl5VvOzwuMiJIwCqfnsA16jLu9\n+HkWyv/SV38pUxbUKfZnv6hIn6DHjCqK6ppkv1DCRDdQi1sZS7c1CqrLUE9EDHFNs7F8kXHJ5gHn\ncKbRADASWskJLxlvt4WlrOlLR9G0VC1SjFb88RYTXgf9HoCRUHasEo1K9czSHIPktQrn1vDdzu4+\ngLYyVy1f9PotJkYa3bufNzRExXBxqIYtPVHr9Yc9AJHyeD2VHCVD7lkTCjaa/3JBJV+KfzXkE5gV\nyXo4HALoaZBGN+ipoGROcL4iisqNxyy0EqrD556/gWlihP3+WJErwOqIrHQmORZcAxAK6V+8dCVT\nrly5CuANaycszsKdnRYVocWPPmIXq6tXObbnnqOyssJU1RklxyKXBzBQW9ZYv4684LyFAi1x1Cpz\nUuOxHIutzxwmi4Qcp7sTJ07+Aoh7YDlx4uTcyBRImETEULPzREzdfgygJ3xhUaTLl0lCcPd9orzD\nnpIDq4wkXiZhN9Y/Wc+UzU3ii5/50msAusId9TXmDc6vMbbyqEnc11NL1UKVNvzsMo//Sp1j2N1j\nDGh9fQNAR7X7LbWWXF6i2V8HjeErNWK35brI0UEcESrGVD2DS+za8y9nyq//g3/EQcYEGp/cY8wu\nyYnEQpHEkTLimi3Vuyc9ALG6ZipGhATEL0dtFuv7T2n2byondihUkigdsaoo5P1PNwA8eKTAq1Ix\nFxYX9F2l6aqR6p4mEEogHDMdSqlVygAaJZ7FOAX7nemxVEzkozb3OOz7YmqPkiGARoOlo2trpBYI\nVao2Cgknk5RDaguJ9/rG5DHUaIWh8h4mcF9J3BJl5YuaTyBRuK0qBkdDZAVV2Nlqz8KpRi6Y80/G\n7Eai4d/YZ+Vmr9vKFCuoXL1wEdPEF1wyBToRcgrsjtMsT9X66SOrQKzP1IHJuk0pRrGvyHu7yfvy\n9h7x4wfv/ihT5lX/u7rKn9vq2hUApZLynBcYWFxaoRvH0nftlkXyMFjr1nHiqOWdJh4mWBzSM5jv\n4SwsJ06cnCOZYmEdiVKgLA/xcBBCnS0wkZi/NM/X9V15zneafAHu6Z3cmOGj9/aLdEnef8iclIzA\nypziN2/Qc3zjKq2y9c1WpmSl5wD294xfQd1f9G7ceJ/m2NZ+G0BOIQJfFf+rF2m4XdFT+rJ8+cZ+\nNZQplyR5DXJ6LcWv/f2/xwGs8p357vuMKpgHNBy3CVG9hVy25lbM+prE9m6xHp/jVwm3hHo37u3R\ngrNUI7OEGmKJylzRzX01Rpe/dm9PjULFFxzJ6W7FOtY5pqK26SUlH3mRDyC0jjRqf1JWatVpaal+\naGuThm1VlT3Pv/AigAWxklUU+hiI3fjggGl3xiTRE61CRXlqs3Wu0qp61pcLeQCBbKVYTvcsyANg\nJKLqgXV2GXP+iqVXeEKZbQj8AoBULVcHQyr7uzQY9/YZX7JqMGPCMF6QokiNT0guNQuLW8xFnZOp\nYtwGExUxVMzn3e/QHt/e2gKwtUWjqX2oCjkZhjMK+9RklJXEO2IUchvbXNJ319kNdzCIAUQxD7K4\nRFR05w7r7W7eYDLa0hJva32WkZNimU+AFFot+oHQpjfabud0d+LEyV8AcQ8sJ06cnBuZAgnv36P5\nd1nJ+yUvBJAIRARmQ5qHT07lmkiBn3+eqTR/+Af/JVN6LVqnlQX6Vu9t7AC4eJH+vKu3SO9bFCR5\n7ln2kmk1W5ny4Yf07icCIxsHtPPbfdn5cRFAu0WkubxKG/VRk1vmLzYyZV+QB+qV0pK/OVVDoEEy\nxDR55503MuW9997JFE+GrifT2koczOcKGCMCjeqg4GFiJgtjogLx+SpFy0v5Ub1IL7UnAozIt2tX\n+lgKAAUhkUgsgL2WRRV0XVasIxQaynsdqW1SRzioUggALIvALxAuK5xZSoH5Zd7ueWEEYwHOFtKR\nCqSOOup4WsxraOLVkxv+mRVGToq6d771jlUxVnfQBzBQsKIlXGmQbTDgGW/fJjdeXhmFE3zBJ1v4\nDLtHADa26dDY2aWvOhSUNnIRyywryFXS0TV+4xvfwFRRMldiOVaR6mOEFq0PVM5X0pMglS83/Ltv\nvckztnYALCiJ7PEWr72uZLG8VrgF2epKPQvEjlIITnpgOl4HwH6LgZr1ByxQax1wWt56QwtYpJiX\nL9MVsyZa8Atr/EmurXBLtTYHIFcW5YN3Zlqfs7CcOHFybsQ9sJw4cXJuZAokfOdTBqEu3yEleYIu\ngJzFy2S1ttXPo9VioGRh/uVM+ZWv/WKmvPx5Wt2//R9/J1NyKvWenZ0D8Mwao2zGue5HDBLNr3J4\na9eICFrCIG+/806mbHXEvZ2nrTu7ughg8Tr/9AXHYpnUd9UI59620rv03LY6la6uNUpsir6FCfmj\nb/+PTOmJGq2Q52HLqkGx6fVTVfZbG8u8QcIcgFLxJMouiF8hUOpZSW1lDWgodgevZC8ehV3CEMBA\nZTEjBcgs88heVcF4i65UkHy2RujRqHBLreIDKAT8Sl4pQrl4OnDGRGcUS9oKBHuTDOxYDYoyniz1\nrSTcN+hy/P1DLrm+uq8GBZtSEcXFEYBPPiRaebi+zpEI+BuSWrvANKJ5kSP2BetMOZA7otnaB9AL\nLf/L6EC4xXr62s2oCFttqbZpW7UyJ2QkhG4h5lwk2gZDi9o5VQqVhRQ7Cg4O+jzOrZsvAHjl5dey\nP994jy0I/vhHzLE6FKl/rLWxvMqQ35e//OVMCXTL1tUs9vs/+D6Az71ALv+6XEA7uq5tJQlaTHZV\nJBBXr17hGRUT7x4d6opSAHm1sx2c4hQxcRaWEydOzo24B5YTJ07OjUyBhHfbBCN7sagL8gMAXij7\nLRH/lrLs1tRQ88tfYqSvlGfc6uqzLPj+q3/71zPl3//Of86U3e1DAJuHRhvA/qwFWbzNHpV7YoOA\nIjLpIpHm3DJHa2AnSxlNSrZdJGRCsoeRijbGiZG0rbsezfuR4l5pMt06XVmiMbzVZ/wljluZUlez\nzEClOe091moctWmHj2KLfw0xtRZBHGaFMufWMG805njj+6ai1q1VkazHWVauHVb8ATkBqJJwX1mT\nMK8U3EtSLq4xJCcgjsHgCICndrOBMEmjXj45fsndTz7MlBfuEEfYtGej8xSaS1TDYbCiJwb6QY8R\nagsXWqPca6IzX15mgmJW+REoVjsr9kQ7ryXlWvLnRx9/kilGUGHOAcuizBZYR0mhfXEWGiQM1avN\nOOMfPeXasAzS+IwGVuO2o2P2dP5vJHlCzEgEEi2oWVY4+Mtf+ao+8TDB137zZbp3XvwpKgqujuff\negVcu8bM7UAzduUGSf7WLt8CUC7zdlufARu/9Rkw3Le8xNRxo3zwhZQ9eWniZAhgpCtNctNnCc7C\ncuLEyTmSKRbWJ2qP+rvfpaPuC88uAlgtqHm3XiAXVvnsvLDIl9hz11TbqRKKrV0+cb/+b2lYvfU2\nX7lZxc9E6YucpnLXxSUeNjY3s/zl1ic18tSI/PilDELrPmJ9t2kn+LI7UtUMR7LO8lY6Y0lJ4fQq\ngXSkEvEq30JH1jUz5kv4+du0KZILtLl29zgbO6Lr7bRiTLylYyVSJRGPVg34Xnr+JfJQb8q5uyt/\nfz88+drPSn+KKq6q6pY1qpyuJTX7WV3jTbyu2uOVEqeuo8SofdXHZh7uquIAtRm+aRcW5nCGjJT0\nNOhwtJ6sy8yKMHos63h67y7tnSMLaOidnFdQwhrfJ1aqbWW9cQpgcYGDNHuqN7aeqDx69PjEPqZY\np/ieCrAPWy0A3T21yw1s2Lwco+jqKk0pUo1RPO7tPt126PdpQvpKHwtEBh3qpxQp9zDSldphjd3M\nqneiOMJEM5tQ1uva5au6QhWHSfFEmvbgETPX+qGhFrFmz16dPN3BofpOaTaq9Su6UNX5H/LSNp82\nNVqOsqj6uayyKFdTdfrBmU2YnIXlxImTcyPugeXEiZNzI1MgYUd22jfeYh3Mp/fuA/grr7Ig+7k1\ngpQH90lD/POvkau3lKer+EiI7Lf/G/M+3v6Axfq9SAUxQQmAJzfwuJeknO6pZz45fjQUZBtpS07J\nNUMdNjM3g+AkuKtUZH/KtI4NQ2ge7ESR2mQWlB12QvY3Wcgej2i+9mXt96zHqjpfLolAKj8kZCuL\nYKHvpwDS1ICxsIP8jr0+weOXXyfAvHObpMyPHjE7Zu+ATn0rE8kc2oabSjrdkiBVQ8xZsc64vcej\nfSJeJMjnWl8mvKrM1gFUZvjdebFr1eR8PS1l3YhQiMxCHFnQxpMzuSDcak16LDJQk1PZU2ZQRRcS\nKWfn7sek62g39wG0VA2TWLtcHS3QkiiJ5MD4LnrC9bsi7TJI6HsBgDmth1B79pQSFokEIhkDwJO0\nCrncdBPh29/+nxx8RMLiipKSElE/G/nHGIQmlhopwmgLESQRAE9IbSBwl4z5sKz+RlGXBmMstZqu\nUT147Dv00Ht2B5UEZwmGenpY0MMbE4+cuvZxUmAMAFUdRIGs0+IsLCdOnJwbcQ8sJ06cnBuZAgkX\nFmkZNptEJVutAwDfU6OaePSs9qXVtyQSO/i02H/4Bin3fv+b38uUYVLVOcVq4B17XMaWYyVD0Zqh\nGieslddYjCZnBSXGkeD5mMj1mDH227H5OjpxtEQkCmZar64S49TrVN7AMVlV4G/jEbFhNLTsGCoP\nFO06VJ6UXXBX6V3daAQgiQ0SiodaIGI4IOJ467v/PVN+scoruqMr6os+wUJmWR3VwCJcAhE7exzt\nQxEW7/UY9hoIHpUEAOeVXlea5fj9cgEChgCKwpU5f8pC4iVrkAZGPNVmZaMd6AItx6pkeTpSLMAX\nNulYeGw0x8ZZbKHeoICJkqygZEfjkELh/Y4oPSxuaB1hLTZc0vhH/RDASIwCFpC1AJ/5NCxzKlKi\nYhob7J0edC5phURCgoFqwoLCrEaiuKQ5T8b83epVYyCRa83CiOHx7ZOXaI2UbA9ROap3VDhQ6dXx\n32ykJrgGzC030NNoPZzEjyZhx259BGCgz0vBHs4QZ2E5ceLk3Ih7YDlx4uTcyBRLPpAdmy+IQmxQ\nBPBgx+okmPn5C6+Qpa/cYEF2W5zov/HHhFAD2agjGfxFMXtlJrTlTJr4MiZPW8/F00jQdlYtTrlU\nxkQmm5GyH6nhipVHDAVSZhus6li9QKUmHNETI8UJuXyTJGTtLiFVd8PsWHG/Ceg1daKCqmpChQXj\njLHbYLAdIrW4Erfce4/x1sdHnMklT+1XlS4Yy+rueAmA7ZRo5Z6ikxtiBegZAcNl1uivXiGbWkm1\nLDCgJ8q9Wq0GoKIonqfE1PSM4BeAtpg8ekoc3dkUB8MgBBArRdYoJUaCbHZd1kI0L6KIcRRYit3x\nbMKMm2HQURxZRAtHh1QsNludUVKxJjAdKTAtFsMsr/VQ3YYMCcbKyTRi+OTU3TSCitwYsh0TyxPu\ndBjwrcjFYceyTsAWCgwjG5syLT2LG0YAQmPpEPeDJZ2Ow4WG2cdI08alORT+zb5lZXATFWXxCcUa\nqXqnfsf2USAgORpFAHpzXFcXLs7gDHEWlhMnTs6NuAeWEydOzo1MbaRqPT5lKwYlAGFEu3ynQ/vz\nzY8ZsvmVHm28o5QAarNJpaQgXdSzHDYRhFcqAAIZq1Yfn9OoLKxgMcFUHAbGhGfFZZ2Qww6jLgQM\nMVH+bgCwO6ChWxMSnFtmPZ2RiH/8gCHRfGK27jGpNxhKW1phKG1LkNAsYKvMH8pOtp5RsXo3xTgJ\nH04M2w43Egbp7jGtzis2MsUX68CmTvQOhgDuCUB1apy32kUW/S2qbe2irr2o5MwxF5+gTVEM9H7g\nA/CtyaiF87TltGw//FQHO1kBl0XTAjG4Ww/OnHUzzRMWVUSOmDuFXyIVhHYiRRKHEYBElXFjAjz1\n+yoUGYlbfoaT0BVcbR9QsaZn6TgKmcMEgZ+xOKTpyTtl2DBvRAu6y73edA/DxiPSDX66zfNWldQa\nCEVG45XFGYv1UaKgc36chj2CKgoB6NLHLgZrEGtd+8YxR9tH99dmO2OkSOKT8VBPvo6cGErG5PRa\nRafmCR3l9MZzFQDPvMQmEnUlFJwWZ2E5ceLk3Mi09JkxZY96cnh5AIn1mJSZs77D18XXf5utcb76\nlVcz5f4mrYDeONdJvnxVV/iFAoCK3pkF2Up9FVWYvzyVcZQv8dS+3vnmoLUt2aO9b3k6uhzboSHj\naGGVsYLdPdaRt1SV0npEu+D6NVW3H5eSOtYUdTn2covlr5XfHFHu5JSOi/azneztY/vpLZdK6egt\n95Fe8rNqqPPxgKzW74tdulmvAJi/xMGvXSErWWON116ocPzWhHWksQWya3wpgd722Rt1bCLl7AV7\n5pvPT5SmFJu7VzZLdjRL2EntLc3vDsW8HIkbI9GcTvAfUMzpnnUVNesgkKkVaxWViqpYUu3RwR7t\nmq5iLHmtdt+6ew6HmOhhYybweBK0kq3jaUlLriPaiV73ENPEg1aRLYTYiKRlAdkk+5pJGVDWHtWK\nsbI5tilNlftmk5sadDCCCuvBI/7uWJ+NzJTz8wBS61Rk5F1mnVnb1/H8KI6h8EgkMuu6mEIuvngT\nQJDj7WjdfR9niLOwnDhxcm7EPbCcOHFybmQKJJxXU0mrmehGIYCCOi9aKoenRK3v/PC9TFnfpBv+\nsEcbuykPvTJCUBUYySoMijqI4Y6S/OW+zHL7yGzUSEAvZ749mbhxOMJEBkpZSHNxntQC80tEgqHA\nwlB1/H1j7xWgyLpynhYrlO+qWH+mwRMNugQyxv0QyyqOxwa/xm8W9HFJhX1SJUN1lWLzXXFVPxKF\n9H5FuUgrzA5bvbgE4NoiowoLCi94mvyuAKDVQwTywtaMOVo7B0qdK5UrAIqa0rzIOT5DzNU9ZgFW\n+lOa5ACkikSMkaa+ay722Nz8QqnFohwLWiTG+pDy4OZv1u1Q1CJU+lhP6UXd4zUiAHIFHnagPMFs\n/FoyY0xvkNC2GBtEGvLUB/vE7KPwjOVkpJXaYSSsbh9BxTqWg5gIf3maWyPqS9IIkzBcnpmCrt3w\nZZIeQ+iYyMMajeSrNy97mmKiFe6YhcI8C6nc/7mTP9WRqC7nbpKC+eJVlvQNnu4A+LH4NsoipDwt\nzsJy4sTJuRH3wHLixMm5kSmQcCgQpE4rGCYjAHlxM0RmFRv/gUJm65vkALA6+yg0y9a646hZaa+D\nidiKlexUraFLhdjQk8FZMP42oRWrvN9tijEaESZKN+aU1LG60KCyykhZS9it3WI9REfdTRrqfLO3\nM71wPFQxhF+gxTu3xBON1H80UrhwZJE4494WJMyuzDJ3cqeCg1BVRyDeu1FZpS2zHORzsysaNqtq\navUAwIygYlGVRgOr6rB4pdHaiT9vjBY0hrwgeRZpzWtPS8hKz6AqBzAIrfTfIlbH0nw8XaCRu9uS\nmIB7wiCWPWSwKzm5wLJamZHSCX2t55FwX6zDVrUUDQl6RpLRV7HL8T43yal4riVkBQLINi3Np/w5\njIaM3uZOQn+JBQBF5+ApXphXTA3x+IfHnRV5H5M2mIshzQEoCdg26mK4twvRsM25YT+Zgm732Pmj\n72WRRPtK58hi8Tqs7mbbgs6LPPXlmzczZX6eDoqNj9goa//efUxknJUKZ02Ts7CcOHFyfsQ9sJw4\ncXJuZCokpDFcFAbJCvsTxS9yFqSwou2xchIJpslJ0z0dl3onmLD/Dw6I6ZpiSa/XWJAxK4BW92hM\nJurtGSVDXYkgQMkHMBTlmDGIB7KWo566MPW4T6fF7luJyAyM7XsQTH+mB0oTbSwQnNbESB0PxWom\nKGgNoNIxmZlRC3iYQCLWm9aI0AJBzrKyEOs1VZbUSO1WE1NFTbA6s+pDcfJ1NNqeAQHjNRcrQEGI\nzACgAbEx/kpTAKGK7AsFKUo1PC35ovE1KnPYPAmehwmmh3FwcJxmezKwCEUSLQJrlWSRQloZsX1f\nSDDuq5hGUcKqvlKeZeDY+OdGKtvyToE3onWL/I4bnlKrCq522/QwtJUvaojZ7jtUlcLt5mcx8nWV\nSKXim/RVkWOKDXLM22d1NrkUE5yIvaCtAYxBob4rX42mJQjtbnqnvnVMIo3Nzmvp5fVlFYHduqZj\n8UQf//AHmTLc4e/Oj2NMVAslZ7SbhbOwnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJ\nEyf/z8r/Ab+8NWulkQjIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7FFA9E5C8F28>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next() # 一个batch返回4张图片\n",
    "print('实际的label: ', ' '.join(\\\n",
    "            '%08s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid(images / 2 - 0.5)).resize((400,100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着计算网络预测的label："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "预测结果:   frog   car  ship  ship\n"
     ]
    }
   ],
   "source": [
    "# 计算图片在每个类别上的分数\n",
    "outputs = net(images)\n",
    "# 得分最高的那个类\n",
    "_, predicted = t.max(outputs.data, 1)\n",
    "\n",
    "print('预测结果: ', ' '.join('%5s'\\\n",
    "            % classes[predicted[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经可以看出效果，准确率50%，但这只是一部分的图片，再来看看在整个测试集上的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000张测试集中的准确率为: 53 %\n"
     ]
    }
   ],
   "source": [
    "correct = 0 # 预测正确的图片数\n",
    "total = 0 # 总共的图片数\n",
    "\n",
    "\n",
    "# 由于测试的时候不需要求导，可以暂时关闭autograd，提高速度，节约内存\n",
    "with t.no_grad():\n",
    "    for data in testloader:\n",
    "        images, labels = data\n",
    "        outputs = net(images)\n",
    "        _, predicted = t.max(outputs, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum()\n",
    "\n",
    "print('10000张测试集中的准确率为: %d %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练的准确率远比随机猜测(准确率10%)好，证明网络确实学到了东西。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  在GPU训练\n",
    "就像之前把Tensor从CPU转到GPU一样，模型也可以类似地从CPU转到GPU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.3732)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = t.device(\"cuda:0\" if t.cuda.is_available() else \"cpu\")\n",
    "\n",
    "net.to(device)\n",
    "images = images.to(device)\n",
    "labels = labels.to(device)\n",
    "output = net(images)\n",
    "loss= criterion(output,labels)\n",
    "\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果发现在GPU上并没有比CPU提速很多，实际上是因为网络比较小，GPU没有完全发挥自己的真正实力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对PyTorch的基础介绍至此结束。总结一下，本节主要包含以下内容。\n",
    "\n",
    "1. Tensor: 类似Numpy数组的数据结构，与Numpy接口类似，可方便地互相转换。\n",
    "2. autograd/: 为tensor提供自动求导功能。\n",
    "3. nn: 专门为神经网络设计的接口，提供了很多有用的功能(神经网络层，损失函数，优化器等)。\n",
    "4. 神经网络训练: 以CIFAR-10分类为例演示了神经网络的训练流程，包括数据加载、网络搭建、训练及测试。\n",
    "\n",
    "通过本节的学习，相信读者可以体会出PyTorch具有接口简单、使用灵活等特点。从下一章开始，本书将深入系统地讲解PyTorch的各部分知识。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
